Skip to content

What happens if first record does not have headers but further records in the same batch do? #227

Open
@jeqo

Description

@jeqo

Output schema (avro/parquet) is based on the first record. This works fine for key/value, etc. but headers may differ between records. Different record header elements types can cause issues but this can be solved by casting header types by using StringConverter or similar for headers.
Though, when first record does not have headers the schema type is null. We should figure out what happens with next records in the same batch if they do have header (are the headers missing? does conversion fail?) and provide an approach on how to process infer the schema for headers properly.

See Aiven-Open/gcs-connector-for-apache-kafka#347 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions