Experiencing issue with AWS MSK IAM between Proxy and Brokers, and SASL Plain between Client and Kafka Proxy

[AnkitMishra3220]

I am using Amazon MSK with IAM enabled and trying to connect to the Kafka Proxy using the SASL Plain authentication mechanism from my Kafka Client.

I am getting an error "using plain connection instead of TLS " in the proxy log while trying to connect from Kafka client.

Getting below error on Kafka Client :

terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)

Kafka Proxy Command :

/kafka-proxy server --bootstrap-server-mapping "b-1-XXXX.amazonaws.com:9098,xtssff.com:9000" \
                   --bootstrap-server-mapping "b-2.XXXX.amazonaws.com:9098,xtssff:9001" \
                   --bootstrap-server-mapping "b-3.XXXX.amazonaws.com:9098,xtssff:9003" \
                   --proxy-listener-key-file "server-key.pem" \
                   --proxy-listener-key-password "XXXXXXX" \
                   --proxy-listener-cert-file "server-cert.pem" \
                   --proxy-listener-ca-chain-cert-file "ca.pem" \
                   --proxy-listener-tls-enable \
                   --auth-local-param "--username=abc" \
                   --auth-local-param "--password=XXXX"
                   --sasl-enable \
                   --sasl-method "AWS_MSK_IAM" \
                   --sasl-aws-region "us-east-1" \
                   --log-level debug    

Kafka Client Config File

ssl.truststore.location=/home/ec2-user/kafka_2.13-3.5.1/bin/kafka.client.truststore.jks
sasl.mechanism=SCRAM-SHA-512
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="abc" \
  password="XXXX";    

I have used AWS Certificate Manager to generate the certificate and below command to generate kafka.client.truststore.jks file :

/usr/lib/jvm/jre-11-openjdk/bin/keytool -import \
	-trustcacerts \
	-noprompt \
	-alias test \
	-file server-cert.pem \
	-keystore ./kafka.client.truststore.jks \
    -storepass changeit 

Could you please help me in resolving this issue ?