[FLINK-38886] Introduce GenericRecordData and Internal / External converters in flink-cdc-common#4218
Merged
Merged
Conversation
Member
Author
|
Updated this PR to reflect newly introduced Variant type, PTAL. @lvyanquan |
…onverters in flink-cdc-common
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces GenericRecordData and a set of converter classes to provide a portable way to construct and manipulate CDC RecordData without depending on heavy runtime packages. The converters facilitate bidirectional conversion between CDC internal objects and Java objects, simplifying connector implementation.
Changes:
- Added
GenericRecordDataclass for generic RecordData implementation backed by Object arrays - Introduced converter classes:
CommonConverter,InternalObjectConverter,InternalClassConverter,JavaObjectConverter, andJavaClassConverter - Added VARIANT type support to
SchemaMergingUtilsand related tests - Created documentation for type mappings between CDC internal types and Java external types
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| GenericRecordData.java | New generic RecordData implementation using Object array storage |
| GenericArrayData.java | Added toString() method for better debugging support |
| CommonConverter.java | Shared conversion logic between Java and internal objects |
| InternalObjectConverter.java | Converts Java objects to CDC internal representations |
| InternalClassConverter.java | Maps DataTypes to CDC internal class types |
| JavaObjectConverter.java | Converts CDC internal objects to Java objects |
| JavaClassConverter.java | Maps DataTypes to Java class types |
| SchemaMergingUtils.java | Added VARIANT type coercion support |
| SchemaMergingUtilsTest.java | Extended tests to include VARIANT type |
| InternalObjectConverterTest.java | Test coverage for internal object conversions |
| InternalClassConverterTest.java | Test coverage for internal class mappings |
| JavaObjectConverterTest.java | Test coverage for Java object conversions |
| JavaClassConverterTest.java | Test coverage for Java class mappings |
| VariantConvertingTest.java | Test coverage for VARIANT type conversions |
| type-mappings.md (EN/ZH) | Documentation for type mapping reference |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Mrart
pushed a commit
to Mrart/flink-cdc
that referenced
this pull request
Mar 26, 2026
…onverters in flink-cdc-common (apache#4218)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This closes FLINK-38886.
Currently, there's no portable way to construct a CDC RecordData without depending on heavy flink-cdc-runtime packages. Also BinaryRecordData is not easy to parse and inspect.
Providing such classes and converters between CDC Internal objects / classes and Java objects / classes would simplify connectors implementation.