Commit bd6084c
feat: Add GaussDB CDC connector with critical bug fixes
## Summary
Add complete GaussDB CDC connector implementation with support for snapshot and streaming modes using GaussDB's mppdb_decoding logical replication plugin.
## Key Features
- Full snapshot support for initial data capture
- Streaming CDC using GaussDB logical replication
- Support for all common GaussDB data types
- Configurable connection pooling and retry mechanisms
- Comprehensive test suite
## Critical Bug Fixes
### 1. Default Value Converter Issue
- **Problem**: GaussDB returns function calls (e.g., `pg_systimestamp()`, `CURRENT_TIMESTAMP`) as default values, causing Debezium to fail when trying to use them as actual values
- **Solution**: Created `GaussDBDefaultValueConverter` that properly handles function-based defaults by returning `Optional.empty()` for them
- **Files**:
- `io/debezium/connector/gaussdb/GaussDBDefaultValueConverter.java` (new)
- `io/debezium/connector/gaussdb/GaussDBSchema.java` (modified)
### 2. Missing Source Info Fields
- **Problem**: Debezium envelope requires multiple fields (version, connector, name, snapshot) in the source struct, but they were not being set, causing validation errors
- **Solution**: Added all required source info fields to both snapshot and streaming source struct builders
- **Files**:
- `GaussDBScanFetchTask.java:294-339` (snapshot)
- `GaussDBStreamFetchTask.java:495-551` (streaming)
## Implementation Details
### Core Components
- **GaussDBSource**: Main source implementation
- **GaussDBDialect**: Dialect for GaussDB-specific SQL and behavior
- **GaussDBConnection**: Connection management with retry logic
- **GaussDBReplicationConnection**: Logical replication connection handling
- **GaussDBScanFetchTask**: Snapshot data reading
- **GaussDBStreamFetchTask**: Streaming CDC data reading
### Configuration
- Hostname, port, database, username, password
- Plugin name (mppdb_decoding)
- Slot name for logical replication
- Connection timeout and retry settings
- Snapshot fetch size
### Testing
- Unit tests for all major components
- Integration tests for snapshot and streaming modes
- Data type compatibility tests
- Boundary condition tests
## Verified Configuration
- ✅ GaussDB `wal_level = logical` (required for CDC)
- ✅ `mppdb_decoding` plugin available and functional
- ✅ Replication slot creation and management working
## Known Issues
- Integration test timeout issue under investigation (fetch task execution)
- Requires further debugging of Flink job initialization flow
## Dependencies
- GaussDB JDBC driver (included in lib/)
- Debezium PostgreSQL connector (for replication protocol)
- Flink CDC base framework
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>1 parent c1a7d0b commit bd6084c
66 files changed
Lines changed: 16742 additions & 42 deletions
File tree
- flink-cdc-connect/flink-cdc-source-connectors
- flink-connector-debezium/src/main/java/io/debezium/relational
- flink-connector-gaussdb-cdc
- docker
- sql
- src
- main
- java
- io/debezium/connector/gaussdb
- connection
- decoder
- org/apache/flink/cdc/connectors/gaussdb
- source
- config
- fetch
- offset
- utils
- table
- resources/META-INF/services
- test
- java
- io/debezium/connector/gaussdb/connection
- org/apache/flink/cdc/connectors/gaussdb
- connection
- source
- config
- fetch
- utils
- table
- resources/ddl
- flink-sql-connector-gaussdb-cdc
- src/main/resources/META-INF
- gaussdb-resource
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 47 additions & 42 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
47 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
48 | 75 | | |
49 | 76 | | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
66 | 81 | | |
67 | 82 | | |
68 | 83 | | |
| |||
72 | 87 | | |
73 | 88 | | |
74 | 89 | | |
75 | | - | |
| 90 | + | |
| 91 | + | |
76 | 92 | | |
77 | 93 | | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
| 94 | + | |
| 95 | + | |
88 | 96 | | |
89 | 97 | | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
94 | 103 | | |
95 | 104 | | |
96 | 105 | | |
97 | | - | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
98 | 109 | | |
99 | 110 | | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
| 111 | + | |
| 112 | + | |
108 | 113 | | |
109 | 114 | | |
110 | 115 | | |
| |||
0 commit comments