Skip to content

Commit e3c67de

Browse files
committed
[avro] Support decimal logical type on fixed base (#798)
Confluent's Avro converter maps both bytes- and fixed-based Avro decimals to a Kafka Connect Decimal (a BigDecimal). The connector already validates (validateDataSchema), converts (StructToJsonMap) and serializes (doWriteColValue) a Connect Decimal to a ClickHouse Decimal(P, S) column, so the fixed base is handled the same as the already-supported bytes base. - Add AvroDecimalLogicalTest: models the Connect representation the converter emits for a decimal-on-fixed field and drives it through the convert + RowBinary write path, asserting the encoded bytes match writeDecimal (no live ClickHouse needed). - Promote decimal_fixed_logical.json from the incompatible to the compatible Avro fixtures so the integration test exercises it end to end. - CHANGELOG entry under 1.4.1 (unreleased).
1 parent 9a48af8 commit e3c67de

3 files changed

Lines changed: 73 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,11 @@
99
The record serialization loop is now shared between the V1 and V2 insert paths. Since client V2 transmits
1010
the INSERT statement as an HTTP query parameter, feature test coverage was added for table names containing
1111
URL-special characters (space, `+`, `&`, `=`, `%`, `?`, `#`), SQL quotes, and non-ASCII characters.
12+
* Support the Avro `decimal` logical type on a `fixed` base (in addition to `bytes`). Confluent's Avro
13+
converter maps both representations to a Kafka Connect `Decimal` (a `BigDecimal`), which the connector
14+
already validates, converts and serializes to a ClickHouse `Decimal(P, S)` column. This promotes the
15+
`decimal_fixed_logical` schema from the incompatible to the compatible Avro test fixtures and adds
16+
feature-test coverage. (https://github.com/ClickHouse/clickhouse-kafka-connect/issues/798)
1217

1318
# 1.4.0, 2026-07-15
1419

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
package com.clickhouse.kafka.connect.sink.db;
2+
3+
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
4+
import static org.junit.jupiter.api.Assertions.assertEquals;
5+
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
6+
import static org.junit.jupiter.api.Assertions.assertNotNull;
7+
8+
import com.clickhouse.data.format.BinaryStreamUtils;
9+
import com.clickhouse.kafka.connect.sink.data.Data;
10+
import com.clickhouse.kafka.connect.sink.data.StructToJsonMap;
11+
import com.clickhouse.kafka.connect.sink.db.mapping.Column;
12+
import com.clickhouse.kafka.connect.sink.db.mapping.Type;
13+
import com.clickhouse.kafka.connect.util.jmx.SinkTaskStatistics;
14+
import java.io.ByteArrayOutputStream;
15+
import java.math.BigDecimal;
16+
import java.util.Map;
17+
import org.apache.kafka.connect.data.Decimal;
18+
import org.apache.kafka.connect.data.Schema;
19+
import org.apache.kafka.connect.data.SchemaBuilder;
20+
import org.apache.kafka.connect.data.Struct;
21+
import org.junit.jupiter.api.Test;
22+
23+
/**
24+
* Reproduction/verification for the Avro {@code decimal} logical type on a {@code fixed} base
25+
* (issue #798, from #726). Confluent's {@code AvroData} converts both {@code bytes}- and
26+
* {@code fixed}-based Avro decimals into a Kafka Connect {@link Decimal} logical field whose value
27+
* is a {@link BigDecimal}; the {@code fixed} case additionally carries a {@code connect.fixed.size}
28+
* parameter. This test models that Connect representation and drives it through the convert path
29+
* ({@link StructToJsonMap}) and the RowBinary write path, without needing a live ClickHouse.
30+
*/
31+
public class AvroDecimalLogicalTest {
32+
33+
@Test
34+
public void decimalOnFixed_convertsToBigDecimal_andSerializesToDecimal_18_4() throws Exception {
35+
// What the Confluent Avro converter emits for {"type":"fixed","size":8,
36+
// "logicalType":"decimal","precision":18,"scale":4}: a Connect Decimal(scale=4) field
37+
// (BYTES base, logical name org.apache.kafka.connect.data.Decimal) + connect.fixed.size,
38+
// value = BigDecimal.
39+
Schema amountSchema =
40+
Decimal.builder(4).parameter("connect.fixed.size", "8").optional().build();
41+
Schema recordSchema =
42+
SchemaBuilder.struct().field("id", Schema.INT32_SCHEMA).field("amount", amountSchema).build();
43+
44+
BigDecimal amount = new BigDecimal("0.0100"); // unscaled 100, scale 4 — matches fixture row 1
45+
Struct struct = new Struct(recordSchema).put("id", 1).put("amount", amount);
46+
47+
// Convert side: the field must survive as a BigDecimal.
48+
Map<String, Data> data = StructToJsonMap.toJsonMap(struct);
49+
Data amountData = data.get("amount");
50+
assertNotNull(amountData, "amount field should be converted");
51+
assertInstanceOf(BigDecimal.class, amountData.getObject(), "decimal must convert to BigDecimal");
52+
assertEquals(amount, amountData.getObject());
53+
54+
// Write side: into a ClickHouse Decimal(18, 4) column.
55+
Column col = Column.extractColumn("amount", "Decimal(18, 4)", false, false, false);
56+
assertEquals(Type.Decimal, col.getType());
57+
assertEquals(18, col.getPrecision());
58+
assertEquals(4, col.getScale());
59+
60+
ClickHouseWriter writer = new ClickHouseWriter(new SinkTaskStatistics(0));
61+
ByteArrayOutputStream out = new ByteArrayOutputStream();
62+
writer.doWriteColValue(col, out, amountData, false);
63+
64+
ByteArrayOutputStream expected = new ByteArrayOutputStream();
65+
BinaryStreamUtils.writeDecimal(expected, amount, 18, 4);
66+
assertArrayEquals(expected.toByteArray(), out.toByteArray(), "RowBinary decimal encoding must match");
67+
}
68+
}

src/testFixtures/avro/schemas/incompatible/decimal_fixed_logical.json renamed to src/testFixtures/avro/schemas/compatible/decimal_fixed_logical.json

File renamed without changes.

0 commit comments

Comments
 (0)