adding fix for koperator to add interBrokerListenerName one at a time and check for security.inter.broker.protocol #101
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hvan
reviewed
Feb 27, 2025
hvan
approved these changes
Feb 27, 2025
added 2 commits
February 27, 2025 14:18
hvan
approved these changes
Feb 27, 2025
dvaseekara
added a commit
that referenced
this pull request
Jun 4, 2025
* Add koperator/api changes for KRaft support * Move processRoles under brokerConfig * Update comments for ZK-relevant configurations in kafkacluster_types.go * Rebase origin/kraft-api * Support running Kafka cluster in KRaft mode * brokerRoles -> processRoles to match upstream Kafka naming * Update pod start-up process so pods can be restarted during rolling upgrade triggered by controller addition/removal * Rebase from origin/kraft-api * Update exsiting integration tests and func signatures * Fix broker configurations; add unit tests for broker configurations under KRaft mode * Remove unnecessary method from koperator/api * Extend integration tests to cover KRaft mode * make lint-fix * Update static KafkaCluster yamls; add check for kraft mode before setting ClusterID in status * Rebase from origin/koperator-api * Use util functions that got moved to the koperator/api module * Remove unineteded changes during rebase * Do not take active controller identity into consideration when reorder the brokers * Update implementation to accomomdate the latest KafkaCluster API change * Make comments about ZK-relevant configurations more clear * Add ConcurrentBrokerRestartCountPerRack to RollingUpgradeConfig (banzaicloud#1002) * Add ConcurrentBrokerRestartCountPerRack to RollingUpgradeConfig * Small refactoring * Exclude controller-only nodes from all CC operations * Add processRoles label key for Kafka pods in KRaft mode; export consts * Allow concurrent broker restarts from same AZ (broker rack) (#62) * Fix flaky test by deleting nodeports explicitly (#67) * Allow dashes when parsing broker rack (#68) * Upgrade Kafka to 3.6.0 (#69) * Upgrade dependencies * Fix wrong port on expectEnvoyWithConfigAz2Tls test (#70) * Upgrade Kafka to 3.6.1 (#71) Co-authored-by: Petruț™ <[email protected]> * working kraft * Merge origin/master * Fixing go.mod * Fixing tests post merge * Fix tests * Fix tests * Fix unnecessary append * Fix go.mod * Update CRD * Update go.mod for e2e * Update go.mod * More fixes * Upgrade Kafka image to use Java v21 (#72) * Remove jbod cc tests * Merging Master to Kraft and Reverting Rack Removal (#89) * Fix flaky test by deleting nodeports explicitly (#67) * Upgrade Kafka to 3.6.0 (#69) * Upgrade dependencies * Fix wrong port on expectEnvoyWithConfigAz2Tls test (#70) * Upgrade Kafka to 3.6.1 (#71) Co-authored-by: Petruț™ <[email protected]> * Upgrade Kafka image to use Java v21 (#72) * Added arm64 to docker build platforms (#73) * Added arm64 to docker build platforms * Regenerated headers for 2024 * Upgrading Kafka to 3.7.0 (#77) * Update codeql-analysis.yml (#78) * [INTERNAL] Create uniq leader ID per operator deployment (#76) * [INTERNAL] Get watched namespaces from env variable (#75) (cherry picked from commit de6500b) * [CORE-106517] Fix outdated config in the sample (#83) * Cross-compile koperator for arm and intel. (#84) * Adding Contour Ingress support (#82) * Allow property security-inter-broker-protocol (#85) * adding the ability to use security-inter-broker-protocol in koperator * updating util.go to remove _ for generated names * adding replace all for external listener port name * fixing other places where externallistener name is used to not have _ * adding an alternative way to identify which port to use for kafka administration and cc connection * taking out comments for pr push * fixing kafka crd * setting omitempty so it will not be required * adding generated crds * adding comments with context for new flag UsedForKafkaAdminCommunication * Use getBrokerReadOnlyConfig function to get properties and update unit test - security_inter_broker_protocol_Set * Update crds to match generated manifest --------- Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: Ha Van <[email protected]> * Revert "Allow concurrent broker restarts from same AZ (broker rack) (#62)" This reverts commit 514fa07. * Fixed build issues * Fix TestGenerateBrokerConfig * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added watch namesapces * Added tmate for debugging * Added tmate for debugging * Added tmate for debugging * Added enabled projectcontour helm install * Enabled cloud-provider-kind * Added ProjectContour cluster role * updated certificate name * updated certificate name * Run without SSL * Removing Project Contour * Adding cloud-provider-kind * Removing cloud-provider - manually adding during test * trigger test * Remove SnpshotClusterAndCompare * Increased log length for Snapshot and Compare * Re-Add Snapshot and compare * Increased log length for Snapshot and Compare * Increased log length even more * Add Uninstall Contour CRDs * Re-Add KafkaCluster_SSL Tests * Removing BanzaiCloud Helm Chart from list of repos * pushing up latest go.sum * Clean up Merge * Enabling Tmate to debug e2e Test * Revert Cert Changes * Revert "Revert Cert Changes" This reverts commit 5c5b19c. * Enable sslClientAuth * trigger test * WIP: Fix Listener Config * Clean up test case results - tc-1 * Clean up test case results - tc-2 * Updated Kraft Test Cases * Cleanup Linting Issues * Remove Tmate Debugger * Run Kraft CLuster E2E * Add kraft e2e test * Revert Test --------- Co-authored-by: ctrlaltluc <[email protected]> Co-authored-by: Adi Muraru <[email protected]> Co-authored-by: Razvan Dobre <[email protected]> Co-authored-by: Cristian-Petrut Petrache <[email protected]> Co-authored-by: Petruț™ <[email protected]> Co-authored-by: Adrian Muraru <[email protected]> Co-authored-by: Adrian <[email protected]> Co-authored-by: aguzovatii <[email protected]> Co-authored-by: cawright-rh <[email protected]> Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: Ha Van <[email protected]> Co-authored-by: Daniel Vaseekaran <[email protected]> * Trigger E2E Test * End to end testing with KRaft cluster (#92) * Fix flaky test by deleting nodeports explicitly (#67) * Upgrade Kafka to 3.6.0 (#69) * Upgrade dependencies * Fix wrong port on expectEnvoyWithConfigAz2Tls test (#70) * Upgrade Kafka to 3.6.1 (#71) Co-authored-by: Petruț™ <[email protected]> * Upgrade Kafka image to use Java v21 (#72) * Added arm64 to docker build platforms (#73) * Added arm64 to docker build platforms * Regenerated headers for 2024 * Upgrading Kafka to 3.7.0 (#77) * Update codeql-analysis.yml (#78) * [INTERNAL] Create uniq leader ID per operator deployment (#76) * [INTERNAL] Get watched namespaces from env variable (#75) (cherry picked from commit de6500b) * [CORE-106517] Fix outdated config in the sample (#83) * Cross-compile koperator for arm and intel. (#84) * Adding Contour Ingress support (#82) * Allow property security-inter-broker-protocol (#85) * adding the ability to use security-inter-broker-protocol in koperator * updating util.go to remove _ for generated names * adding replace all for external listener port name * fixing other places where externallistener name is used to not have _ * adding an alternative way to identify which port to use for kafka administration and cc connection * taking out comments for pr push * fixing kafka crd * setting omitempty so it will not be required * adding generated crds * adding comments with context for new flag UsedForKafkaAdminCommunication * Use getBrokerReadOnlyConfig function to get properties and update unit test - security_inter_broker_protocol_Set * Update crds to match generated manifest --------- Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: Ha Van <[email protected]> * Revert "Allow concurrent broker restarts from same AZ (broker rack) (#62)" This reverts commit 514fa07. * Fixed build issues * Fix TestGenerateBrokerConfig * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added watch namesapces * Added tmate for debugging * Added tmate for debugging * Added tmate for debugging * Added enabled projectcontour helm install * Enabled cloud-provider-kind * Added ProjectContour cluster role * updated certificate name * updated certificate name * Run without SSL * Removing Project Contour * Adding cloud-provider-kind * Removing cloud-provider - manually adding during test * trigger test * Remove SnpshotClusterAndCompare * Increased log length for Snapshot and Compare * Re-Add Snapshot and compare * Increased log length for Snapshot and Compare * Increased log length even more * Add Uninstall Contour CRDs * Re-Add KafkaCluster_SSL Tests * Removing BanzaiCloud Helm Chart from list of repos * pushing up latest go.sum * Clean up Merge * Enabling Tmate to debug e2e Test * Revert Cert Changes * Revert "Revert Cert Changes" This reverts commit 5c5b19c. * Enable sslClientAuth * trigger test * WIP: Fix Listener Config * Clean up test case results - tc-1 * Clean up test case results - tc-2 * Updated Kraft Test Cases * Cleanup Linting Issues * Remove Tmate Debugger * Run Kraft CLuster E2E * Increate Timeout to allow pod termination * Trigger Test * Added Debugger * Fix App Labels for Controllers * Revert image upate * Revert "Fix App Labels for Controllers" This reverts commit a3cf8a5. * Include Broker/Controller Labels for Headless SVC Selector * Logic for controller listener * add controller service * Added Headless-Controller-SVC Labels * Fix controller addresses and labels for brokers * Empty commit to trigger e2e * Set up kafka-3 as controller only for troubleshooting * Empty commit to trigger e2e * Use controller address for JMXTemplate * Update uninstall timeout to 600s * fix lint * fix lint * fix lint * Enable TMate Debugger * Trigger E2E * Updated BrokerIdLabelkey * Updated BrokerIdLabelkey * Check for Kraft mode when setting the controller listener * Check for Kraft mode when setting the controller listener * Disable tmate from e2e test * adding //nolint:unparam to testProduceConsumeInternal func now that it is used twice * moving the //nolint:unparam to the none-ssl version * Fixed BrokerLabel Test * Add additional test cases for TestGetBrokerLabels * Add additional test cases for TestGetBrokerLabels * commenting out broker-1 in test to fix kraft test * adding conditional to check if kraft mode is enabled before selecting which expected results in test --------- Co-authored-by: ctrlaltluc <[email protected]> Co-authored-by: Adi Muraru <[email protected]> Co-authored-by: Razvan Dobre <[email protected]> Co-authored-by: Cristian-Petrut Petrache <[email protected]> Co-authored-by: Petruț™ <[email protected]> Co-authored-by: Adrian Muraru <[email protected]> Co-authored-by: Adrian <[email protected]> Co-authored-by: aguzovatii <[email protected]> Co-authored-by: cawright-rh <[email protected]> Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: Ha Van <[email protected]> Co-authored-by: Daniel Vaseekaran <[email protected]> Co-authored-by: Ha Van <[email protected]> * [CORE-119212] - Add/fix pdb for controllers (#94) * Ensure externallisteners are listed first (#97) * Exclude controllers from external ingress (e.g. envoy) (#95) * adding new check for broker level readOnlyConfig (#96) * Zk kraft migration (#100) * Allow setting CLUSTER_ID as env var for zk to kraft migration * Add migration properties to control if broker are in kraft mode or zk mode even when kraft is enabled * Refactor functions and additional test cases * Fix linting * Fix unit test * Add zkConnect property when broker is in zk mode * Remove control.plane.listener.name property from broker during migration * Add additional labels to brokers for backward compatibility while performing migration * adding fix for koperator to add interBrokerListenerName one at a time and check for security.inter.broker.protocol (#101) * adding fix for koperator to add interBrokerListenerName one at a time * updating test to include inter.broker.listener.name * fixing code that checks for inter.broker.listener.name * removing unecessary function calls * updating test expectations to be alphabetized * adding generateListenerSpecificConfig back * adding merges back in * reverting back to having getConfigProperties only call generateListenerSpecificConfig * removing unnecessary comments --------- Co-authored-by: Cameron Wright <[email protected]> * [CORE-122420] - Update pod name for Kraft Controllers (#102) * [CORE-122420] - Update pod name for Kraft Controllers * Triggering Integration Tests * Update Test Case --------- Co-authored-by: Daniel Vaseekaran <[email protected]> * Cleanup Merge * Cleanup Merge * Update .github/workflows/helm.yml Co-authored-by: Razvan Dobre <[email protected]> * Update api/go.mod Co-authored-by: Razvan Dobre <[email protected]> * Fixing controller reconcile logic to properly check for controller only nodes in kraft mode only (#104) * fixing controller reconcile logic that was stopping all reconciles * fixing lint issue * fixing filteredBrokerID changes * Updating spacing in cctask_controller_test * increasing timeout time during waitforclusterrunningstate * testing longer pull rate * increasing serviceIP range to try to increase test stability * reverting to 50ms * adding service counter to try and further debug * temporarily removing counter * removing commented section * dummy commit to kick of test * Adding if statement to check if kafkacluster is deleted first before deleting in test * adding additional checks and defering if an error arises to help with debugging * adding further debugging and using a helper function to clean up kafkacluster resources * adding new util to help with safer deleting, easier debugging, and an additional helper function for port management * taking out old tests * adding liscense text to new test_utils.go * taking out old comments from previous testing * fixing lint errors * handling errors * adding panic handling for problematic tests * adding defer to exact spot where panic is occuring * removing safekafkacleanup * removing cleanups * adding logging to help with debug * adding printf formatting directive * adding removal for kafkacluster kraft specific test * changing kfakcluster deletion * taking out manual kraft cleanup * removing manual cleanup of kafkacluster * refactoring externalnodeport test to be more stable * fixing license issue * removing logging * changing logging * getting rid of old comments * updating test to reserve all ports in a line * fixing lint issue * fixing lint issues * Update controllers/tests/suite_test.go Co-authored-by: hvan <[email protected]> * fixing lint errors * forcing ordering to further prevent nodeport collision --------- Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: hvan <[email protected]> * Update api/v1beta1/kafkacluster_types.go Co-authored-by: Razvan Dobre <[email protected]> * Update api/v1beta1/kafkacluster_types.go Co-authored-by: Razvan Dobre <[email protected]> * Update config/samples/banzaicloud_v1beta1_kafkacluster.yaml Co-authored-by: Razvan Dobre <[email protected]> * [CORE-126251] - PR Comments Addressed * [CORE-126251] - PR Comments Addressed --------- Co-authored-by: Darren Lau <[email protected]> Co-authored-by: Lucian Ilie <[email protected]> Co-authored-by: Adi Muraru <[email protected]> Co-authored-by: Cristian-Petrut Petrache <[email protected]> Co-authored-by: Petruț™ <[email protected]> Co-authored-by: Adrian Muraru <[email protected]> Co-authored-by: dvaseekara <[email protected]> Co-authored-by: Adrian <[email protected]> Co-authored-by: aguzovatii <[email protected]> Co-authored-by: cawright-rh <[email protected]> Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: Ha Van <[email protected]> Co-authored-by: Daniel Vaseekaran <[email protected]> Co-authored-by: Ha Van <[email protected]> Co-authored-by: hvan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Please provide a meaningful description of what this change will do, or is for. Bonus points for including links to
related issues, other PRs, or technical references.
Note that by not including a description, you are asking reviewers to do extra work to understand the context of this
change, which may lead to your PR taking much longer to review, or result in it not being reviewed at all.
Type of Change
Checklist