[Avro] Record Union

[chernser]

Description

record unions (see: src/testFixtures/avro/schemas/incompatible/union_two_records.json)

Avro allows to have:

{"name": "payload", "type": [{
        "type": "record",
        "name": "TypeA",
        "fields": [
          {"name": "label", "type": "string"}
        ]
      }, {
        "type": "record",
        "name": "TypeB",
        "fields": [
          {"name": "count", "type": "int"}
        ]
      }]

The only matching type in ClickHouse is JSON because it has no restrictions on value schema.
Dynamic and Variant is not a match for complex objects.

[29shivam]

Hi @chernser — starting on this (record union → JSON, per your guidance). Two things I'd like your call on before I implement, since both affect the stored value:

1. JSON value shape. For payload as a union of two records, the connector currently serializes the Confluent union struct in tagged form, keyed by the branch record name — e.g. {"TypeA": {"label": "foo"}} / {"TypeB": {"count": 42}}. Into a ClickHouse JSON column, do you want:

• (a) that tagged form (preserves which record type it was), or
• (b) the unwrapped branch content, i.e. {"label": "foo"} / {"count": 42} (loses the type tag, and TypeA/TypeB could collide on same-named fields)?

I lean (a) since it's lossless, but it's your call.

2. Fixture target + integration config. The union_two_records fixture currently targets Variant(Tuple(...), Tuple(...)); I'll switch it to JSON per your note. But writing to a ClickHouse JSON column in RowBinary needs input_format_binary_read_json_as_string=1 plus the connector's JSON-as-string setting, and the Avro integration harness (setupAvroConnector) doesn't set those today — no existing compatible fixture targets JSON. How would you like that handled: add those settings to the Avro connector config used by the parameterized test, or a per-fixture config hook?

Happy to implement as soon as you point me at (1) and the config approach for (2).

[29shivam]

Gentle nudge on this one @chernser — no rush at all. I've got #798 and #799 up (PRs #802/#803) and I'm ready to implement this record-union → JSON piece as soon as you can weigh in on the two calls above: the JSON value shape (tagged {"TypeA": {...}} vs unwrapped {...}) and how you'd like the JSON-as-string config wired into the Avro integration harness. Whenever you get a moment.

[chernser]

@29shivam
Thanks! I will look this week.

[chernser]

@29shivam

Item 1
Yes, today's implementation with tag works fine. We have not got any request to do something else. ClickHouse works great with JSON - so it should be no problem for processing.

Item 2
Hmm... Looks like we need to add setupAvroConnectorWithJson and set needed property. Because we also need to verify correct behavior when input_format_binary_read_json_as_string=0 and JSON is present. It is two different branches.

[29shivam]

Thanks @chernser, that's clear. Plan:

Item 1 — value shape: keep the current tagged form ({"TypeA": {...}}) as-is, no serialization change.

Item 2 — integration: add setupAvroConnectorWithJson that sets the JSON-as-string property, retarget union_two_records from Variant(Tuple, Tuple) to JSON, and route it through that setup. I'll cover both branches you called out:

• input_format_binary_read_json_as_string=1 → the record union writes into the JSON column (happy path).
• input_format_binary_read_json_as_string=0 with JSON present → today the connector throws Writing JSON in binary is not supported yet…; I'll add coverage asserting that behavior — let me know if you'd rather it be handled differently (e.g. routed to DLQ) instead of failing the task.

Will get the PR up shortly.