Migrate source mssql from old CDK to new CDK #63731

burakku · 2025-07-23T07:40:26Z

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

vercel · 2025-07-23T07:40:31Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)			Jul 23, 2025 7:40am

github-actions · 2025-07-23T07:40:50Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

/format-fix - Fixes most formatting issues.
/bump-version - Bumps connector versions.
- You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
- Leaving the changelog arg blank will auto-populate the changelog from the PR title.
/run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
/build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
/poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
/poe source example lock - Alias for /poe connector source-example lock.
/poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
/poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

github-actions · 2025-07-23T07:45:48Z

`source-mssql-v2` Connector Test Results

0 tests 0 ✅ 0s ⏱️
0 suites 0 💤
0 files 0 ❌

Results for commit 7b41d01.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-02T22:57:44Z

`destination-dev-null` Connector Test Results

47 tests 15 ✅ 39s ⏱️
4 suites 32 💤
4 files 0 ❌

Results for commit fc52150.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-02T22:58:20Z

`destination-customer-io` Connector Test Results

48 tests 18 ✅ 25s ⏱️
7 suites 30 💤
7 files 0 ❌

Results for commit fc52150.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-02T22:59:22Z

`destination-hubspot` Connector Test Results

44 tests 14 ✅ 25s ⏱️
5 suites 30 💤
5 files 0 ❌

Results for commit fc52150.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-02T23:01:28Z

`source-mysql` Connector Test Results

8 files 8 suites 27s ⏱️
51 tests 51 ✅ 0 💤 0 ❌
175 runs 175 ✅ 0 💤 0 ❌

Results for commit fc52150.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-02T23:01:33Z

`destination-mssql` Connector Test Results

135 tests 117 ✅ 7m 32s ⏱️
12 suites 18 💤
12 files 0 ❌

Results for commit fc52150.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-02T23:09:45Z

`destination-s3` Connector Test Results

808 tests 459 ✅ 2h 34m 35s ⏱️
26 suites 349 💤
26 files 0 ❌

Results for commit 66ff966.

github-actions · 2025-09-03T18:15:54Z

Deploy preview for airbyte-docs ready!

✅ Preview
https://airbyte-docs-atejcwfpu-airbyte-growth.vercel.app

Built with commit b3afefe.
This pull request is being automatically deployed with vercel-action

github-actions · 2025-09-23T08:15:17Z

`source-mssql` Connector Test Results

7 files 7 suites 14m 8s ⏱️
61 tests 61 ✅ 0 💤 0 ❌
171 runs 171 ✅ 0 💤 0 ❌

Results for commit b3afefe.

♻️ This comment has been updated with latest results.

github-actions · 2025-10-02T18:36:01Z

Note

Detected that there are differences in the Gradle dependencies.

https://github.com/airbytehq/airbyte/runs/52742686365

rodireich · 2025-10-09T21:20:47Z

airbyte-integrations/connectors/source-mssql/metadata.yaml

Progressive rollout needs also

releases: rolloutConfiguration: enableProgressiveRollout: true

rodireich · 2025-10-09T21:24:42Z

...s/source-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlSourceOperations.kt

+    ) {
+        recordData.set<JsonNode>(
+            CommonMetaField.CDC_UPDATED_AT.id,
+            CdcOffsetDateTimeMetaFieldType.jsonEncoder.encode(timestamp),


This should match the change in MsSqlServerDebeziumOperations:
CommonMetaField.CDC_UPDATED_AT.type.jsonEncoder as JsonEncoder<Any>

That's when we decorate NativeRecordPayload not ObjectNode right?

ohhh 🤦
that's right.
ignore

github-actions · 2025-10-10T23:25:36Z

docs/integrations/sources/mssql.md

-| 4.2.3 | 2025-07-01 | [62052](https://github.com/airbytehq/airbyte/pull/62052) | Revert change to CDK interface signature. |
-| 4.2.2 | 2025-06-25 | [61729](https://github.com/airbytehq/airbyte/pull/61729) | Support the use of logical primary keys for CDC. |
-| 4.2.1 | 2025-06-23 | [62015](https://github.com/airbytehq/airbyte/pull/62015) | Fix previous merge. Improve cutoff date handling |
+| 4.3.0   | 2025-09-26 | [63731](https://github.com/airbytehq/airbyte/pull/63731) | Migrate source mssql from old CDK to new CDK |


🚫 [vale] _{reported by reviewdog 🐶}
[Vale.Spelling] Did you really mean 'mssql'?

github-actions · 2025-10-10T23:25:36Z

docs/integrations/sources/mssql.md

-| 4.2.2 | 2025-06-25 | [61729](https://github.com/airbytehq/airbyte/pull/61729) | Support the use of logical primary keys for CDC. |
-| 4.2.1 | 2025-06-23 | [62015](https://github.com/airbytehq/airbyte/pull/62015) | Fix previous merge. Improve cutoff date handling |
+| 4.3.0   | 2025-09-26 | [63731](https://github.com/airbytehq/airbyte/pull/63731) | Migrate source mssql from old CDK to new CDK |
+| 4.2.5   | 2025-08-13 | [64905](https://github.com/airbytehq/airbyte/pull/64905) | bumping up java cdk version for mssql |


🚫 [vale] _{reported by reviewdog 🐶}
[Vale.Spelling] Did you really mean 'cdk'?

github-actions · 2025-10-10T23:25:36Z

docs/integrations/sources/mssql.md

-| 4.2.2 | 2025-06-25 | [61729](https://github.com/airbytehq/airbyte/pull/61729) | Support the use of logical primary keys for CDC. |
-| 4.2.1 | 2025-06-23 | [62015](https://github.com/airbytehq/airbyte/pull/62015) | Fix previous merge. Improve cutoff date handling |
+| 4.3.0   | 2025-09-26 | [63731](https://github.com/airbytehq/airbyte/pull/63731) | Migrate source mssql from old CDK to new CDK |
+| 4.2.5   | 2025-08-13 | [64905](https://github.com/airbytehq/airbyte/pull/64905) | bumping up java cdk version for mssql |


🚫 [vale] _{reported by reviewdog 🐶}
[Vale.Spelling] Did you really mean 'mssql'?

github-actions · 2025-10-10T23:25:36Z

docs/integrations/sources/mssql.md

-| 4.2.1 | 2025-06-23 | [62015](https://github.com/airbytehq/airbyte/pull/62015) | Fix previous merge. Improve cutoff date handling |
+| 4.3.0   | 2025-09-26 | [63731](https://github.com/airbytehq/airbyte/pull/63731) | Migrate source mssql from old CDK to new CDK |
+| 4.2.5   | 2025-08-13 | [64905](https://github.com/airbytehq/airbyte/pull/64905) | bumping up java cdk version for mssql |
+| 4.2.4   | 2025-07-03 | [62491](https://github.com/airbytehq/airbyte/pull/62491) | Improve Debezium performance by configuring the poll interval parameter. |


🚫 [vale] _{reported by reviewdog 🐶}
[Vale.Spelling] Did you really mean 'Debezium'?

mwbayley

Some questions and small requests. This is a huge piece of work! Thanks for handling it.

mwbayley · 2025-10-09T21:22:35Z

.../main/kotlin/io/airbyte/integrations/source/mssql/MsSqlServerCdcInitialSnapshotStateValue.kt

+    @JsonProperty("pk_val") val pkVal: String? = null,
+    @JsonProperty("pk_name") val pkName: String? = null,
+    @JsonProperty("version") val version: Int? = null,
+    @JsonProperty("state_type") val stateType: String? = null,
+    @JsonProperty("incremental_state") val incrementalState: JsonNode? = null,
+    @JsonProperty("stream_name") val streamName: String? = null,
+    @JsonProperty("cursor_field") val cursorField: List<String>? = null,
+    @JsonProperty("stream_namespace") val streamNamespace: String? = null,


shouldn't the default naming strategy for Jackson convert these names to snake_case?

I looked this up and it's configurable in the ObjectMapper. We don't do that in Jsons, however. I imagine it's because we want to avoid doing any kind of renaming of fields or namespaces in the connection state. Given that, this way seems ideal.

mwbayley · 2025-10-09T21:29:25Z

.../main/kotlin/io/airbyte/integrations/source/mssql/MsSqlServerCdcInitialSnapshotStateValue.kt

+            primaryKey: List<Field>,
+            primaryKeyCheckpoint: List<JsonNode>,
+        ): OpaqueStateValue {
+            val primaryKeyField = primaryKey.first()
+            return when (primaryKeyCheckpoint.first().isNull) {
+                true -> Jsons.nullNode()
+                false ->
+                    Jsons.valueToTree(
+                        MsSqlServerCdcInitialSnapshotStateValue(
+                            pkName = primaryKeyField.id,
+                            pkVal = primaryKeyCheckpoint.first().asText(),


Why are we only using the first PK field? Imagine the PK fields are year, month, day. This way, we can't checkpoint until we finish reading a whole year's data, right? Is the snapshot state value object defined for backwards compatibility with old states so we can't add the other fields/values?

mwbayley · 2025-10-09T22:01:55Z

...e-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlServerDebeziumConverter.kt

+        private const val MSSQL_DATE_TYPE = "DATE"
+        private const val MSSQL_DATETIME_TYPE = "DATETIME"
+        private const val MSSQL_DATETIME2_TYPE = "DATETIME2"
+        private const val MSSQL_SMALLDATETIME_TYPE = "SMALLDATETIME"
+        private const val MSSQL_DATETIMEOFFSET_TYPE = "DATETIMEOFFSET"
+        private const val MSSQL_TIME_TYPE = "TIME"
+        private const val MSSQL_SMALLMONEY_TYPE = "SMALLMONEY"
+        private const val MSSQL_MONEY_TYPE = "MONEY"
+        private const val MSSQL_BINARY_TYPE = "BINARY"
+        private const val MSSQL_VARBINARY_TYPE = "VARBINARY"
+        private const val MSSQL_IMAGE_TYPE = "IMAGE"
+        private const val MSSQL_GEOMETRY_TYPE = "GEOMETRY"
+        private const val MSSQL_GEOGRAPHY_TYPE = "GEOGRAPHY"
+        private const val MSSQL_UNIQUEIDENTIFIER_TYPE = "UNIQUEIDENTIFIER"
+        private const val MSSQL_XML_TYPE = "XML"
+        private const val MSSQL_HIERARCHYID_TYPE = "HIERARCHYID"
+        private const val MSSQL_SQL_VARIANT_TYPE = "SQL_VARIANT"


as far as I can tell these are each used once - maybe we can just inline them?

mwbayley · 2025-10-09T22:06:13Z

...e-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlServerDebeziumConverter.kt

+        if (value == null) return null
+
+        return try {
+            val formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSSSS")


this format is also present in MsSqlServerCursorCutoffTimeProvider. Maybe we can centralize the definition?

mwbayley · 2025-10-09T22:07:45Z

...e-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlServerDebeziumConverter.kt

+        if (value == null) return null
+
+        return try {
+            val formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSSSSXXX")


This one looks slightly different than the one in MsSqlServerCursorCutoffTimeProvider. Should it be?

mwbayley · 2025-10-14T04:46:35Z

...s/source-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlSourceOperations.kt

+            is And -> conj.flatMap { it.bindings() }
+            is Or -> disj.flatMap { it.bindings() }
+            is WhereClauseLeafNode -> {
+                val type = column.type as LosslessJdbcFieldType<*, *>


This is related to some of the type issues that Rodi and I have been looking at recently. It would be nice if we had compile-time enforcement of which field types were eligible for use as a cursor and which weren't. Probably nothing we can do about it at the moment.

mwbayley · 2025-10-14T04:53:03Z

...s/source-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlSourceOperations.kt

+        // For MSSQL, we would need to deserialize the state to get the LSN
+        // This is a placeholder implementation - actual implementation would extract LSN from state


Looks like this is what we're doing. Old comment?

mwbayley · 2025-10-14T04:53:22Z

...s/source-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlSourceOperations.kt

+                }
+            }
+        } catch (e: Exception) {
+            // Log error but don't fail the sync


looks like we're not logging anything ATM

mwbayley · 2025-10-14T04:54:19Z

...s/source-mssql/src/main/kotlin/io/airbyte/integrations/source/mssql/MsSqlSourceOperations.kt

+                    }
+                }
+            } catch (e: Exception) {
+                // Log error but don't fail the sync - keep the empty string value


same - no logging currently

mwbayley · 2025-10-14T05:00:06Z

airbyte-integrations/connectors/source-mssql/src/main/resources/application.yml

+          - pattern: (?i).*timeout expired.*
+            input-example: >-
+              com.microsoft.sqlserver.jdbc.SQLServerException: The query has timed out.


I imagine this was copied from the old connector, but it looks like the input example doesn't contain the pattern.

octavia-squidington-iii added the connectors/source/mssql-v2 label Jul 23, 2025

burakku force-pushed the wenqi-mssql-migration branch 2 times, most recently from 9a4ae16 to 66ff966 Compare September 2, 2025 22:53

burakku force-pushed the wenqi-mssql-migration branch from 2e95eb8 to 003eeae Compare September 3, 2025 18:10

burakku force-pushed the wenqi-mssql-migration branch 3 times, most recently from a3b9731 to b0d1572 Compare September 5, 2025 23:56

burakku force-pushed the wenqi-mssql-migration branch 2 times, most recently from f07ec67 to 7b41d01 Compare September 10, 2025 23:18

burakku mentioned this pull request Sep 11, 2025

[source-mssql] Azure Entra authentication #61547

Open

2 tasks

burakku changed the title ~~Wenqi mssql migration~~ Migrate source mssql from old CDK to new CDK Sep 11, 2025

octavia-squidington-iii added the connectors/source/mssql label Sep 23, 2025

burakku force-pushed the wenqi-mssql-migration branch from e44a5f6 to fc52150 Compare September 24, 2025 04:22

burakku force-pushed the wenqi-mssql-migration branch from fc52150 to 0da601c Compare October 2, 2025 18:31

github-actions bot added the dependencies-change label Oct 2, 2025

burakku force-pushed the wenqi-mssql-migration branch 2 times, most recently from 98efdc7 to 392fbf9 Compare October 8, 2025 22:56

burakku marked this pull request as ready for review October 8, 2025 22:58

burakku requested a review from a team October 8, 2025 22:59

burakku force-pushed the wenqi-mssql-migration branch 5 times, most recently from 652f675 to 15df445 Compare October 9, 2025 17:14

rodireich reviewed Oct 9, 2025

View reviewed changes

burakku force-pushed the wenqi-mssql-migration branch from 5d1a565 to dd22c3e Compare October 9, 2025 22:18

rodireich approved these changes Oct 9, 2025

View reviewed changes

burakku force-pushed the wenqi-mssql-migration branch 2 times, most recently from 6d1670a to 9735e06 Compare October 10, 2025 23:23

github-actions bot reviewed Oct 10, 2025

View reviewed changes

mwbayley reviewed Oct 14, 2025

View reviewed changes

burakku force-pushed the wenqi-mssql-migration branch from 8db7e49 to 84192a3 Compare October 14, 2025 06:01

burakku added 12 commits October 14, 2025 12:36

mssql full refresh

7b74864

mssql cdc

ac8c467

mssql fix and tests

c5b6850

dbz heartbeat handling

72a7791

speed mode

f427d6b

hack

9bd927b

Clean up

4b48a08

Fix type

e2b38e6

Fix speed mode

ab9a90b

Fix test

59b7163

Fix tunnel session for dbz

9323725

Fix event_serial_no metadata

b3afefe

burakku force-pushed the wenqi-mssql-migration branch from 84192a3 to b3afefe Compare October 14, 2025 19:43

		// For MSSQL, we would need to deserialize the state to get the LSN
		// This is a placeholder implementation - actual implementation would extract LSN from state

Migrate source mssql from old CDK to new CDK #63731

Are you sure you want to change the base?

Migrate source mssql from old CDK to new CDK #63731

Conversation

burakku commented Jul 23, 2025

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

Uh oh!

vercel bot commented Jul 23, 2025

Uh oh!

github-actions bot commented Jul 23, 2025

👋 Greetings, Airbyte Team Member!

Helpful Resources

PR Slash Commands

Uh oh!

github-actions bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-mssql-v2 Connector Test Results

Uh oh!

github-actions bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

destination-dev-null Connector Test Results

Uh oh!

github-actions bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

destination-customer-io Connector Test Results

Uh oh!

github-actions bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

destination-hubspot Connector Test Results

Uh oh!

github-actions bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-mysql Connector Test Results

Uh oh!

github-actions bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

destination-mssql Connector Test Results

Uh oh!

github-actions bot commented Sep 2, 2025

destination-s3 Connector Test Results

Uh oh!

github-actions bot commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-mssql Connector Test Results

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

mwbayley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 23, 2025 •

edited

Loading

`source-mssql-v2` Connector Test Results

github-actions bot commented Sep 2, 2025 •

edited

Loading

`destination-dev-null` Connector Test Results

github-actions bot commented Sep 2, 2025 •

edited

Loading

`destination-customer-io` Connector Test Results

github-actions bot commented Sep 2, 2025 •

edited

Loading

`destination-hubspot` Connector Test Results

github-actions bot commented Sep 2, 2025 •

edited

Loading

`source-mysql` Connector Test Results

github-actions bot commented Sep 2, 2025 •

edited

Loading

`destination-mssql` Connector Test Results

`destination-s3` Connector Test Results

github-actions bot commented Sep 3, 2025 •

edited

Loading

github-actions bot commented Sep 23, 2025 •

edited

Loading

`source-mssql` Connector Test Results

github-actions bot commented Oct 2, 2025 •

edited

Loading

mwbayley Oct 14, 2025 •

edited

Loading