[avro] Support record unions as JSON (#800) - #814
Open
29shivam wants to merge 1 commit into
Open
Conversation
Confluent's Avro converter turns a union of two records (e.g. [TypeA, TypeB]) into a Connect union struct (io.confluent.connect.avro.Union) keyed by branch record name. Per the maintainer, the value is kept in this tagged form (which branch it was is preserved); a ClickHouse JSON column stores it as-is. Writing to a JSON column in RowBinary requires input_format_binary_read_json_as_string=1, so the Avro integration harness now routes fixtures whose target table has a JSON column through a new setupAvroConnectorWithJson connector config. - Add clickhouse_sink_avro_json.json (Avro converter + JSON-as-string setting) and SinkConfigs.AVRO_JSON / setupAvroConnectorWithJson; avroSchemaTest routes to it when a fixture targets a JSON column. - Add AvroRecordUnionJsonTest: asserts the tagged-by-branch JSON serialization of a record union (no live ClickHouse needed). - Retarget union_two_records from Variant(Tuple, Tuple) to JSON and promote it from the incompatible to the compatible Avro fixtures. - CHANGELOG entry under 1.5.0.
29shivam
requested review from
BentsiLeviav,
chernser,
mshustov and
mzitnik
as code owners
August 5, 2026 02:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #800 (part of #726). Implements the direction from your comment @chernser.
Confluent's Avro converter turns a union of two records (
[TypeA, TypeB]) into a Connect union struct (io.confluent.connect.avro.Union) keyed by branch record name. Per your Item 1, the value is kept in this tagged form (which branch it was is preserved) — no serialization change, a ClickHouseJSONcolumn stores it as-is.Per your Item 2, writing to a
JSONcolumn in RowBinary needsinput_format_binary_read_json_as_string=1, so the Avro integration harness now routes JSON-target fixtures through a dedicated connector config.Changes:
clickhouse_sink_avro_json.json+SinkConfigs.AVRO_JSON+setupAvroConnectorWithJson— Avro converter with the JSON-as-string setting.avroSchemaTestroutes to it when a fixture's target table has aJSONcolumn.AvroRecordUnionJsonTest— asserts the tagged-by-branch JSON serialization of a record union (parses the output; no live ClickHouse needed).union_two_recordsfromVariant(Tuple, Tuple)→JSONand promote it fromincompatible/→compatible/.1.5.0.On the two branches you mentioned: this covers the
input_format_binary_read_json_as_string=1(happy) path end to end. For the=0+ JSON-present branch — today the connector throwsWriting JSON in binary is not supported yet…I'd like your call on whether that should stay a task failure or route to DLQ before I add that coverage (happy to do it in this PR or a follow-up).Verification: feature test passes locally the end-to-end Avro→JSON path runs via the integration test in CI.
Checklist
Delete items not relevant to your PR: