Getting some issues when trying to connect to shema registry

[elidrism]

Hi,

I'm trying to connect to the schema registry in order to deserialize avro format. I'm using spark 2.4.7 with scala_2.11.

Here's my schema registry configuration:

val registryConfig = Map(
          AbrisConfig.SCHEMA_REGISTRY_URL -> "schema_registry_url",
          "schema.registry.ssl.truststore.location" -> "truststore.jks",
          "schema.registry.ssl.truststore.password" -> "*******",
          "schema.registry.ssl.keystore.location" -> "keystore.jks",
          "schema.registry.ssl.keystore.password" -> "******",
          "schema.registry.ssl.key.password" -> "*******",
          "key.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer",
          "value.deserializer" -> "io.confluent.kafka.serializers.KafkaAvroDeserializer",
          "specific.avro.reader" -> "true"
        )

and I'm getting this error:

INFO SchemaManagerFactory: Configuring new Schema Registry instance of type 'io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient'
java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

here's some dependencies in my pom.xml:

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
            <version>2.3.0</version>
</dependency>

<dependency>
            <groupId>za.co.absa</groupId>
            <artifactId>abris_2.11</artifactId>
            <version>4.2.0</version>
 </dependency>

I also set the spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore... in my spark configurations.

Thanks for your help !!

[cerveada]

I don't think this is an Abris issue, rather some ssl configuration problem.

Googling the exception got me this:
https://stackoverflow.com/questions/43387692/ssl-java-security-nosuchalgorithmexception

[kevinwallimann]

Please be aware that the the Schema Registry dedicated client configuration properties (like schema.registry.ssl.truststore.location) are only available since Confluent platform 5.4, see https://docs.confluent.io/5.4.2/schema-registry/security/index.html#additional-configurations-for-https. For older versions, you need to set all ssl flags as JVM options on the driver and executor

spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore=truststore.jks -Djavax.net.ssl.keyStore=keystore.jks -Djavax.net.ssl.trustStorePassword=*** -Djavax.net.ssl.keyStorePassword=*** -Djavax.net.ssl.password=***
spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=truststore.jks -Djavax.net.ssl.keyStore=keystore.jks -Djavax.net.ssl.trustStorePassword=*** -Djavax.net.ssl.keyStorePassword=*** -Djavax.net.ssl.password=***

[elidrism]

Thank you for your reply, but as I said, I already set all ssl flags. here's what I set in my spark-submit configuration:

--conf "spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore=truststore.jks -Djavax.net.ssl.trustStorePassword=*** -Djavax.net.ssl.keyStore=keystore.jks -Djavax.net.ssl.keyStorePassword=*** -Djavax.net.ssl.password="
--conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=truststore.jks -Djavax.net.ssl.trustStorePassword= -Djavax.net.ssl.keyStore=keystore.jks -Djavax.net.ssl.keyStorePassword=*** -Djavax.net.ssl.password=***" \

But it still cannot recover the key.

And the keystore and truststore are not the issue neither the passwords since I can connect to the kafka brokers

[elidrism]

It seems that having 3 different passwords could be the problem. I still don't know why but when I created new keystore ans truststore with same passwords, it fixed the issue.