Compatibility Apicurio Confluent-Kafka Python libraries.

[juliantxo163]

Hello. I am noob developing with python and I am not sure if it is possible what I am trying to do. I thought that Apicurio Registry was compatible with Confluent Schema Registry, in order to use confluent-kafka-python libraries.

I have deployed an Apicurio registry (last release in memory image , in Kubernetes v1.22.3 minikube).

Everything right till now, I am able to create, update or delete artifacts by the API rest or the UI.

But I am not pretty sure if I can use confluent library python to develop a producer with Apicurio registry as Schema registry. I am trying to run and example from: https://github.com/confluentinc/confluent-kafka-python/blob/master/examples/json_producer.py

(julian) [julian@julianmachine examples]$ python json_producer.py -b localhost:9092 -s http://localhost:8082/apis/registry/v2
Producing user records to topic example_serde_json. ^C to exit.
Enter name: admin  
Enter address: street123
Enter favorite number: 15
Enter favorite color: red
Traceback (most recent call last):
  File "json_producer.py", line 161, in <module>
    main(parser.parse_args())
  File "json_producer.py", line 140, in main
    value=json_serializer(user, SerializationContext(topic, MessageField.VALUE)),
  File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/json_schema.py", line 254, in __call__
    self._normalize_schemas)
  File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 339, in register_schema
    body=request)
  File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 128, in post
    return self.send_request(url, method='POST', body=body)
  File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 177, in send_request
    response.json().get('message'))
confluent_kafka.schema_registry.error.SchemaRegistryError: RESTEASY003210: Could not find resource for full path: http://localhost:8082/apis/registry/v2/subjects/example_serde_json-value/versions?normalize=False (HTTP status code 404, SR code 404)
(julian) [julian@julianmachine examples]$ 

Does I missing some parameters? Maybe a group.id in apicurio named "subjects"...?

[apicurio-bot[bot]]

Thank you for reporting an issue!

Pinging @jsenko to respond or triage.

[forsberg]

The Confluent Python libraries are, as far as I know, built to work with the Confluent Schema Registry API specification.

Apicurio implements a compatibility mode, but you will have to change the API path to http://localhost:8082/apis/ccompat/v7.

It looks like it will only work for schemas which are in the default group, i.e where no group has been assigned in the Apicurio UI.

[EricWittmann]

[juliantxo163]

I have the same error with that URL...

(julian) [julian@julianmachine examples]$ python json_producer.py -b localhost:9092 -s http://localhost:8082/apis/ccompat/v7 Producing user records to topic example_serde_json. ^C to exit. Enter name: julian Enter address: pamplona Enter favorite number: 15 Enter favorite color: red Traceback (most recent call last): File "json_producer.py", line 161, in <module> main(parser.parse_args()) File "json_producer.py", line 140, in main value=json_serializer(user, SerializationContext(topic, MessageField.VALUE)), File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/json_schema.py", line 254, in __call__ self._normalize_schemas) File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 339, in register_schema body=request) File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 128, in post return self.send_request(url, method='POST', body=body) File "/home/julian/.local/share/virtualenvs/julian-DzUCWoBY/lib/python3.7/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 177, in send_request response.json().get('message')) confluent_kafka.schema_registry.error.SchemaRegistryError: RESTEASY003210: Could not find resource for full path: http://localhost:8082/apis/ccompat/v7/subjects/example_serde_json-value/versions?normalize=False (HTTP status code 404, SR code 0)
The only parameter that I set is the localhost:9092 for the brokers and http://localhost:8082/apis/ccompat/v7 for the Apicurio Schema as you have sugested me.

On other hand... do you know any example code or a producer/consumer in python that use the Apicurio Registry Client Python?. Thanks.

[carlesarnal]

Sorry for being late to the party. @juliantxo163 which Apicurio Registry version are you using? The http://localhost:8082/apis/ccompat/v7 path was introduced fairly recently so I recommend giving http://localhost:8082/apis/ccompat/v6 a try.

About the Apicurio Registry Python client, I don't have any examples, but you should be able to use the ccompat API and the confluent libraries.

[juliantxo163]

Hello @carlesarnal thank you very much, it works with the http://localhost:8082/apis/ccompat/v6 path!!.

Do you have in mind any documentation where explain why this path, and the properties that are compatible with Confluent Schema registry and considerations for baring in mind if we use it? I am not able to find really good information about it.

Thanks in advance!!

[carlesarnal]

Hello,

About why this path, there's no documentation on the why, it's just by design. We support a set of different APIs and that is the one for the confluent compatibility API.

About the different options yes, you have more information here -> https://www.apicur.io/registry/docs/apicurio-registry/2.4.x/getting-started/assembly-configuring-the-registry.html#_category_ccompat.