SCRAM-SHA-512 auth fails with Kafka 4.0.0 brokers using the existing images

[brokenjacobs]

Debug isn't particularly helpful, the server response comes back and the SCRAM just fails. I suspect the old librdkafka is using the unversioned SASL auth method that was deprecated or something similar? Kafka 4.0.0 is supposed to be baselined back to 2018, and librdkafka 1.8.2 came out in 2021, but for whatever reason this is broken.

I upgraded to librdkafka v2.10.0 and build the alpine release for the dockerfile with 3.21.3 and SCRAM-SHA-512 works again. I'll open a PR to resolve this issue with the changes.

[mohag]

It would be interesting to see which librdkafka version fixes this - we are having issues with other librdkafka-based clients as well.

What I know so far:
2.3.0 fails, 2.8.0 succeeds (messing with the library version on Ubuntu - the OS repo has 2.3.0, oldest one in the Confluent repo is 2.8.0)

Edit: It seems that 2.6.1 is the version that fixes that error.

[darkedges]

Can confirm this works