Commit 5336d69
Avichay Marciano
[FLINK-39415][postgres] Fix TIMESTAMPTZ type mapping in pipeline connector
Root cause: PostgresTypeUtils (pipeline connector) mapped TIMESTAMPTZ to
ZonedTimestampType (TIMESTAMP_WITH_TIME_ZONE), but the Debezium deserializer
only has a converter for TIMESTAMP_WITH_LOCAL_TIME_ZONE. This mismatch causes
the deserializer to produce LocalZonedTimestampData while BinaryWriter.write()
casts to ZonedTimestampData, resulting in binary data corruption and
NumberFormatException when the Iceberg sink reads the field via
BinaryRecordData.getZonedTimestamp().
The existing test PostgresFullTypesITCase already expects LocalZonedTimestampData
for TIMESTAMPTZ (line 1211: DataTypes.TIMESTAMP_LTZ(0)), confirming the correct
type mapping is TIMESTAMP_WITH_LOCAL_TIME_ZONE.
Changes:
- PostgresTypeUtils: TIMESTAMPTZ -> TIMESTAMP_LTZ(scale) (was ZonedTimestampType)
- DebeziumEventDeserializationSchema: add convertToZonedTimestamp() for future
TIMESTAMP_WITH_TIME_ZONE support
- Add PostgresTypeUtilsTimestamptzTest: validates TIMESTAMPTZ maps to
TIMESTAMP_WITH_LOCAL_TIME_ZONE type root
- Add IcebergTypeUtilsTest: validates TIMESTAMP_LTZ type conversion and
field getter creation for Iceberg sink1 parent 13cd198 commit 5336d69
4 files changed
Lines changed: 128 additions & 3 deletions
File tree
- flink-cdc-connect
- flink-cdc-pipeline-connectors
- flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/utils
- flink-cdc-pipeline-connector-postgres/src
- main/java/org/apache/flink/cdc/connectors/postgres/utils
- test/java/org/apache/flink/cdc/connectors/postgres/utils
- flink-cdc-source-connectors/flink-connector-debezium/src/main/java/org/apache/flink/cdc/debezium/event
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
136 | 138 | | |
137 | 139 | | |
138 | 140 | | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
139 | 163 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | 22 | | |
24 | 23 | | |
25 | 24 | | |
| |||
167 | 166 | | |
168 | 167 | | |
169 | 168 | | |
170 | | - | |
| 169 | + | |
171 | 170 | | |
172 | | - | |
| 171 | + | |
173 | 172 | | |
174 | 173 | | |
175 | 174 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
Lines changed: 30 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
199 | 200 | | |
200 | 201 | | |
201 | 202 | | |
| 203 | + | |
| 204 | + | |
202 | 205 | | |
203 | 206 | | |
204 | 207 | | |
| |||
382 | 385 | | |
383 | 386 | | |
384 | 387 | | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
385 | 415 | | |
386 | 416 | | |
387 | 417 | | |
| |||
0 commit comments