Add support Apache Kafka 4.2.0 #12315
Conversation
|
Using this PR in an early stage to test the Apache Kafka 4.2.0 RCs |
|
The failing tests within [ERROR] Errors:
[ERROR] io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiIT.test
[ERROR] Run 1: KafkaConnectApiIT.test » Completion java.lang.AssertionError:
Expected: is "sink"
but: was "SINK"
[ERROR] Run 2: KafkaConnectApiIT.test » Completion java.lang.AssertionError:
Expected: is "sink"
but: was "SINK"
[ERROR] Run 3: KafkaConnectApiIT.test » Completion java.lang.AssertionError:
Expected: is "sink"
but: was "SINK"seem to be related to a regression in Apache Kafka. I opened a JIRA https://issues.apache.org/jira/browse/KAFKA-20076 and a PR to fix it apache/kafka#21318. I also raised this in the voting thread so I would expect a RC1 to continue testing. |
ppatierno
force-pushed
the
add-support-kafka-4.2.0
branch
from
a45d725 to
ab2a7af
Compare
ppatierno
force-pushed
the
add-support-kafka-4.2.0
branch
from
cfdbb84 to
aa23f8c
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #12315 +/- ##
============================================
- Coverage 75.02% 74.99% -0.03%
Complexity 6660 6660
============================================
Files 373 373
Lines 25385 25391 +6
Branches 3411 3414 +3
============================================
- Hits 19044 19043 -1
- Misses 4954 4957 +3
- Partials 1387 1391 +4
🚀 New features to boost your workflow:
|
|
/gha run pipeline=regression,upgrade |
|
⏳ System test verification started: link The following 10 job(s) will be executed:
Tests will start after successful build completion. |
|
❌ System test verification failed: link |
|
/gha run pipeline=regression,upgrade |
|
⏳ System test verification started: link The following 10 job(s) will be executed:
Tests will start after successful build completion. |
|
❌ System test verification failed: link |
|
/gha run pipeline=regression-fg |
|
⏳ System test verification started: link The following 6 job(s) will be executed:
Tests will start after successful build completion. |
|
❌ System test verification failed: link |
|
❌ System test verification failed: link |
|
The failing tests within io.strimzi.operator.common.model.InvalidResourceException: Kafka test-suite-namespace/cluster-46464747 has invalid spec.kafka.config: ssl.enabled.protocols contains values [TLSv1.1] which are not in the allow │
at io.strimzi.operator.cluster.model.KafkaCluster.validateConfiguration(KafkaCluster.java:581) Are all related to a change made in Kafka 4.2.0 where some broker configurations with LIST type, let's take "ssl.enabled.protocols" : {
"items" : [ ],
"scope" : "PER_BROKER",
"type" : "LIST"
}but then the validation logic in the operator fails because doesn't handle empty lists. The mentioned |
|
/gha run pipeline=regression,upgrade |
|
⏳ System test verification started: link The following 10 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
|
/gha run pipeline=regression-fg |
|
⏳ System test verification started: link The following 6 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
|
/gha run pipeline=regression,upgrade |
|
⏳ System test verification started: link The following 10 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
|
/gha run pipeline=regression-fg |
|
⏳ System test verification started: link The following 6 job(s) will be executed:
Tests will start after successful build completion. |
|
❌ System test verification failed: link |
|
/gha run tests=CruiseControlST#testAutoKafkaRebalanceScaleUpScaleDown |
|
⏳ System test verification started: link The following 1 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
ppatierno
force-pushed
the
add-support-kafka-4.2.0
branch
from
09b7938 to
75ac81b
Compare
|
/gha run pipeline=regression,upgrade |
|
⏳ System test verification started: link The following 10 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
|
/gha run pipeline=regression-fg |
|
⏳ System test verification started: link The following 6 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
Signed-off-by: Paolo Patierno <[email protected]>
Signed-off-by: Paolo Patierno <[email protected]>
Signed-off-by: Paolo Patierno <[email protected]>
Signed-off-by: Paolo Patierno <[email protected]>
Signed-off-by: Paolo Patierno <[email protected]>
Added validation check for non empty values and not duplicates in list parameters Signed-off-by: Paolo Patierno <[email protected]>
Signed-off-by: Paolo Patierno <[email protected]>
Signed-off-by: Paolo Patierno <[email protected]>
rebalancing Signed-off-by: Paolo Patierno <[email protected]>
Signed-off-by: Paolo Patierno <[email protected]>
ppatierno
force-pushed
the
add-support-kafka-4.2.0
branch
from
75ac81b to
02dc2db
Compare
Signed-off-by: Paolo Patierno <[email protected]>
|
/gha run pipeline=regression,upgrade |
|
⏳ System test verification started: link The following 10 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
|
/gha run pipeline=regression-fg |
|
⏳ System test verification started: link The following 6 job(s) will be executed:
Tests will start after successful build completion. |
|
🎉 System test verification passed: link |
3 participants
This PR adds the support for Apache Kafka 4.2.0.
Apart the usual changes, it has a change in:
KafkaConfigModelGeneratorby removing the usage of theMetadataVersionValidatorclass because it was actually removed from Apache Kafka 4.2.0 codebase. Such validator wasn't needed since 4.0.0 (when ZooKeeper was removed) but it was still in the Apache Kafka codebase. It was used as validator for config fields likeinter.broker.protocol.versionandlog.message.format.versionthat doesn't exist anymore since Apache Kafka 4.x.MetadataVersion.fromVersionStringmethod is called with a second booleanunstableFeatureVersionsEnabledparameter. We are passing false for that because we don't want to include non production ready versions checks (i.e. we don't want that in our codebase the 4.3 version can be used as a valid version until it's production ready and release).ConnectClusterused by ITs. Apache Kafka 4.2.0 doesn't allow aplugin.pathto be empty. It can be null (not set) but not empty as we have been setting. The change leaves the possibility to set theplugin.pathbut in general we are not setting it for our tests.ConfigModelto handle the validation for config parameters which are empty lists which now show up in the JSON model after a change in Apache Kafka 4.2.0. It also adds a check for non duplicates and non empty values in such lists. More details in the comment below Add support Apache Kafka 4.2.0 #12315 (comment)CruiseControlST.testCruiseControlRemoveDisksModewas fixed because the log segment deletion after partitions movement is asynchronous and we need to wait for it not just checking the disk size at the end of the rebalancing because the log segment would be still there and test fails (related to https://issues.apache.org/jira/browse/KAFKA-19571 backported into Apache Kafka 4.2.0 RC3)