Skip to content

[avro] Support decimal logical type on fixed base (#798) - #802

Open
29shivam wants to merge 1 commit into
ClickHouse:mainfrom
29shivam:fix/798-avro-decimal-logical
Open

[avro] Support decimal logical type on fixed base (#798)#802
29shivam wants to merge 1 commit into
ClickHouse:mainfrom
29shivam:fix/798-avro-decimal-logical

Conversation

@29shivam

Copy link
Copy Markdown
Contributor

Summary

Fixes #798 (part of #726).

Confluent's Avro converter maps both bytes- and fixed-based Avro decimals to a Kafka Connect Decimal logical type whose value is a BigDecimal. The connector already handles that Connect representation end to end — schema validation (ClickHouseWriter.validateDataSchema), conversion (StructToJsonMap), and RowBinary serialization (doWriteColValue) all treat a Connect Decimal as a ClickHouse Decimal(P, S). The already-compatible decimal_bytes fixture confirms the bytes base; this change confirms the fixed base is handled the same way.

Changes:

  • Add AvroDecimalLogicalTest — models the Connect Decimal (with connect.fixed.size) that the converter emits for a decimal-on-fixed field, drives it through the convert + write path, and asserts the RowBinary bytes match BinaryStreamUtils.writeDecimal(...). Runs without a live ClickHouse.
  • Promote decimal_fixed_logical.json from incompatible/ to compatible/ so the Avro integration test exercises it end to end.
  • CHANGELOG entry under 1.4.1 (unreleased).

Verification: the feature test passes locally; the full end-to-end path (Confluent Platform → schema registry → Avro converter → ClickHouse) runs via the integration test in CI.

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

@29shivam

Copy link
Copy Markdown
Contributor Author

@chernser ready for review — this covers the decimal logical-type sub-issue (#798). The Confluent Avro converter already produces a Connect Decimal for both bytes and fixed bases, so the connector handles it end to end; I added a feature test and promoted the fixture to compatible so integration exercises it.

Confluent's Avro converter maps both bytes- and fixed-based Avro decimals to a
Kafka Connect Decimal (a BigDecimal). The connector already validates
(validateDataSchema), converts (StructToJsonMap) and serializes (doWriteColValue)
a Connect Decimal to a ClickHouse Decimal(P, S) column, so the fixed base is
handled the same as the already-supported bytes base.

- Add AvroDecimalLogicalTest: models the Connect representation the converter emits
  for a decimal-on-fixed field and drives it through the convert + RowBinary write
  path, asserting the encoded bytes match writeDecimal (no live ClickHouse needed).
- Promote decimal_fixed_logical.json from the incompatible to the compatible Avro
  fixtures so the integration test exercises it end to end.
- CHANGELOG entry under 1.4.1 (unreleased).
@29shivam
29shivam force-pushed the fix/798-avro-decimal-logical branch from 6745069 to e3c67de Compare August 3, 2026 00:57
@29shivam 29shivam mentioned this pull request Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[avro] Support Decimal Logical Type

1 participant