support for complex avro schema

[melchoir55]

It appears as if the library does not support complex avro schema. That is, avro schema which contain multiple classes referencing each other.

Take the following avro schema, for example:

[{
    "name": "Order",
    "namespace": "com.questanalytics.core",
    "type": "record",
    "fields": [{
        "name": "uid",
        "type": "int",
        "default": "NONE"
    }]
}, {
    "name": "Customer",
    "namespace": "com.questanalytics.core",
    "type": "record",
    "fields": [{
        "name": "uid",
        "type": "int",
        "default": "NONE"
    } {
        "name": "order",
        "type": "Order",
        "default": "NONE"
    }, {
        "name": "name",
        "type": "string",
        "default": "NONE"
    }]
}]

The above has a Customer who has embedded Order objects. Am I wrong in that this library will not allow a developer to work with a schema like this?

We are looking at the writeup here: https://marcosschroh.github.io/python-schema-registry-client/faust/ and it seems as if the faust integration assumes the schema is limited to a single class.

For example, it isn't clear how the following expression (taken from the writeup linked above) could possibly refer to a complex schema:

class UserModel(faust.Record, serializer='avro_users'):
    first_name: str
    last_name: str

It seems as if it assumes the schema contains one and only one class (User)

[marcosschroh]

Hi @melchoir55

I would say that this is not the correct way to register a nested Avro schema

I would expect something like:

{
    "name": "Customer",
    "namespace": "com.questanalytics.core",
    "type": "record",
    "fields": [
        {
            "name": "uid",
            "type": "int",
            "default": "NONE"
        },
        {
            "name": "order",
            "type": {
                "name": "OrderRecord",
                "namespace": "com.questanalytics.core",
                "type": "record",
                "fields": [{
                    "name": "uid",
                    "type": "int",
                    "default": "NONE"
                }]
            },
            "default": "NONE"
        },
        {
            "name": "name",
            "type": "string",
            "default": "NONE"
        }
    ]
}

With the approach will be possible to register the schema in the Schema Registry. For serialization a record example could be:

{
    "name": "test",
    "uid": 1,
    "order": {
        "uid": 2
    }
}

and would be possible to use it in Faust creating two classes, like this example: https://faust.readthedocs.io/en/latest/userguide/models.html#id5 that use Account and Transfer .

You can also use complex nested schema like this with a map type.

What do you think?

[melchoir55]

I see what you're saying. The real problem here is that it doesn't allow you do describe circular relationships. For example maybe Account embeds Transfer and the reverse is also true. Normally in Avro you handle this by describing schema across files, I just collapsed it into one file for convenience.

It also doesn't let you reuse small parts of schema in multiple contexts. You lose the composability of avro by doing this.

I'll probably end up doing it the way you suggest, which is the way that the confluent schema registry expects it anyway. I do still think it isn't best practice, however. Of course as long as the confluent schema registry works the way it does then the value of supporting schema across multiple files is limited.

[forsberg]

The registry does have something it calls schema references, which may or may not be what you're after:

https://docs.confluent.io/current/schema-registry/serdes-develop/index.html#referenced-schemas