Skip to content

Source-MySQL: Update CDC conversion logic to handle null values for non-null columns by converting them to the default epoch value #67623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Source-MySQL: Update CDC conversion logic to handle null values for non-null columns by converting them to the default epoch value #67623

wants to merge 2 commits into from

Conversation

yardencarmeli
Copy link
Contributor

What

Fixes 9165.

Zero-dates in MySQL will be returned by the JDBC driver as null values, which we populate and send to Debezium. However, if the column is NOT NULL, we get an Invalid value: null used for required field error from Debezium that shuts down the connection.

How

  • Updated MySqlSourceCdcTemporalConverter.kt so that NULL values are no longer sent to Debezium. When we receive a null value from JDBC (indicating a zero-date), we convert it to the EPOCH default date (i.e.1970-01-01T00:00:00.000000).
    The only exception is TIME, since MySQL does allow zero-dates for this type. If we receive a NULL value for TIME, it indicates the value is invalid/corrupt (see https://dev.mysql.com/doc/refman/8.0/en/time.html)
  • Updated MySql doc to indicate the new change.

Review guide

User Impact

  • Zero-dates in DATE, DATETIME, and TIMESTAMP columns now return 1970-01-01T00:00:00.000000 (or equivalent for DATE) instead of NULL in CDC mode.
  • Non-CDC syncs are unaffected (zero-dates still return NULL)

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

@yardencarmeli
Source-MySQL: Update CDC conversion logic to handle null values for n…
3745776 
…on-null columns by converting them to the default epoch value
@github-actions GitHub Actions
Copy link
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
  • JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
    • /bump-bulk-cdk-version type=patch changelog='foo' - Bump the Bulk CDK's version. type can be major/minor/patch.
  • Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

github-actions[bot]
github-actions bot reviewed Oct 11, 2025
View reviewed changes
docs/integrations/sources/mysql.md
| `geometry` | base64 binary string | |
| MySQL Type | Resulting Type | Notes |
| :---------------------------------------- | :--------------------- |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `bit(1)` | boolean | |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'boolean'?

docs/integrations/sources/mysql.md
| :---------------------------------------- | :--------------------- |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `bit(1)` | boolean | |
| `bit(>1)` | base64 binary string | |
| `boolean` | boolean | |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'boolean'?

docs/integrations/sources/mysql.md
| `bit(1)` | boolean | |
| `bit(>1)` | base64 binary string | |
| `boolean` | boolean | |
| `tinyint(1)` | boolean | |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'boolean'?

docs/integrations/sources/mysql.md
| `decimal` | number | |
| `binary` | string | |
| `blob` | string | |
| `date` | string | ISO-8601 datetime string. Zero-date values will be converted to Unix epoch (`1970-01-01`) when using CDC mode to prevent sync failures. In non-CDC syncs, zero-dates will be NULL. If using a zero-date column as a cursor or if the column is NOT NULL, convert to EPOCH. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'datetime'?

docs/integrations/sources/mysql.md
| `decimal` | number | |
| `binary` | string | |
| `blob` | string | |
| `date` | string | ISO-8601 datetime string. Zero-date values will be converted to Unix epoch (`1970-01-01`) when using CDC mode to prevent sync failures. In non-CDC syncs, zero-dates will be NULL. If using a zero-date column as a cursor or if the column is NOT NULL, convert to EPOCH. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Will] Avoid using 'will'.

docs/integrations/sources/mysql.md
| `datetime`, `timestamp` | string | ISO-8601 datetime string. Zero-date values will be converted to Unix epoch (`1970-01-01`) when using CDC mode to prevent sync failures. In non-CDC syncs, zero-dates will be NULL. If using a zero-date column as a cursor or if the column is NOT NULL, convert to EPOCH. |
| `time` | string | ISO 8601 time string. Values are in range between 00:00:00 and 23:59:59. |
| `year` | year string | [Doc](https://dev.mysql.com/doc/refman/8.0/en/year.html) |
| `char`, `varchar` with non-binary charset | string | |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'charset'?

docs/integrations/sources/mysql.md
| `text` | string | |
| `mediumtext` | string | |
| `longtext` | string | |
| `json` | serialized json string | E.g. `{"a": 10, "b": 15}` |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'json'?

docs/integrations/sources/mysql.md
| `text` | string | |
| `mediumtext` | string | |
| `longtext` | string | |
| `json` | serialized json string | E.g. `{"a": 10, "b": 15}` |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Google.Latin] Use 'for example' instead of 'E.g.'.

docs/integrations/sources/mysql.md
| `longtext` | string | |
| `json` | serialized json string | E.g. `{"a": 10, "b": 15}` |
| `enum` | string | |
| `set` | string | E.g. `blue,green,yellow` |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Google.Latin] Use 'for example' instead of 'E.g.'.

docs/integrations/sources/mysql.md
|:------------|:-----------|:-----------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.50.9 | 2025-10-06 | [67151](https://github.com/airbytehq/airbyte/pull/66515) | Fix CDC decorating fields encoding to Protobuf |
| 3.50.10 | 2025-10-10 | []() | Fix CDC sync failures caused by zero-dates in non-nullable columns. |
| 3.50.9 | 2025-10-06 | [67151](https://github.com/airbytehq/airbyte/pull/67151) | Fix CDC decorating fields encoding to Protobuf |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Protobuf'?

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 11, 2025 •
edited
Loading

source-mysql Connector Test Results

  8 files    8 suites   28s ⏱️
 51 tests  51 ✅ 0 💤 0 ❌
175 runs  175 ✅ 0 💤 0 ❌

Results for commit df4e390.

♻️ This comment has been updated with latest results.

@yardencarmeli yardencarmeli self-assigned this Oct 11, 2025
@github-actions GitHub Actions
Copy link
Contributor

Deploy preview for airbyte-docs ready!

✅ Preview
https://airbyte-docs-evu1eg38w-airbyte-growth.vercel.app

Built with commit df4e390.
This pull request is being automatically deployed with vercel-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@yardencarmeli yardencarmeli

Labels

connectors/source/mysql

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yardencarmeli @octavia-squidington-iii