fix(source-mysql): derive unicode interpolation base from sampled boundaries to fix collation error 1267

[Yarden Carmeli (yardencarmeli)]

What

Fixes https://github.com/airbytehq/oncall/issues/13104

Fixes error code 1267 (Illegal mix of collations), which occurred when partitioning text primary keys stored in utf8mb3 (the default for UTF8 in MariaDB) or other legacy character set columns. Previously, string partition boundaries were generated using the max Unicode value (U+10FFFF) as the interpolation base. This could produce utf8mb4 characters even when both source boundaries contained only utf8mb3 characters. MySQL could not compare these generated boundaries with an utf8mb3 column, resulting in a collation mismatch.

How

1. Updated StringInterpolationUtils.unicodeInterpolatedStrings() so that the interpolation base is determined by the highest code point present in the provided bounds. This keeps generated code points within the range of the boundaries and reduces the risk of generating characters unsupported by the column's character set.
2. Added a safeguard for surrogate UTF-16 characters that are invalid (U+D800-U+DFFF). Surrogate code points are not valid characters on their own, although pairs of them can represent valid supplementary characters. To fail safely and avoid generating unpaired surrogates, interpolation is skipped when the maximum code point is U+D800 or higher, and the original bounds are returned.
3. Added tests to StringInterpolationUtilTest verifying that:

• Generated interpolations remain within the code point range represented by the bounds.
• Interpolation is skipped when the range could produce surrogate code points.

Review guide

User Impact

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

• 🛠️ Quick Fixes
  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions, scraping changelog description from the PR title.
    • Bump types: patch (default), minor, major, major_rc, rc, promote.
    • The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
    • The promote type strips the RC suffix to finalize a release.
    • Example: /bump-version type=rc or /bump-version type=minor
  • /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
• ❇️ AI Testing and Review (internal link: AI-SDLC Docs):
  • /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
  • /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
  • /ai-review - AI-powered PR review for connector safety and quality gates.
• 📝 AI Documentation:
  • /ai-docs-review - AI-powered documentation review for PRs with connector changes.
  • /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
• 🚀 Connector Releases:
  • /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
  • /enable-autopilot-rollouts - Enables autopilot progressive rollouts for the modified connector(s) in the PR, remediating "autopilot rollouts not enabled for {connector-name}" auto-merge blockers. Sets defaultRolloutMode: autopilot and enableProgressiveRollout: true, preserving any existing autopilotConfig.
    • Optional args: connector=<CONNECTOR_NAME> (defaults to the modified connectors in the PR), strategy=fast|slow|default (defaults to fast).
    • Example: /enable-autopilot-rollouts or /enable-autopilot-rollouts connector=source-faker strategy=slow
• ☕️ JVM connectors:
  • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
    Example: /update-connector-cdk-version connector=destination-bigquery
• 🐍 Python connectors:
  • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
  • /poe source example lock - Alias for /poe connector source-example lock.
  • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
  • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
• ⚙️ Admin commands:
  • /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
    Example: /force-merge reason="CI is flaky, tests pass locally"

📚 Show Repo Guidance

Helpful Resources

• Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
• Developing Connectors Locally
• Managing Connector Secrets
• On-Demand Regression Tests
• #connector-ci-issues
• #connector-publish-updates
• #connector-build-statuses

📝 Edit this welcome message.

[Yarden Carmeli (yardencarmeli)]

/publish-connectors-prerelease connector=source-mysql

Pre-release Connector Publish Started

Publishing pre-release build for connector source-mysql.
PR: #83239

Pre-release versions will be tagged as {version}-preview.59c53ff
and are available for version pinning via the scoped_configuration API.

View workflow run
Pre-release Publish: SUCCESS ✅

Docker image (pre-release):
airbyte/source-mysql:3.53.2-preview.59c53ff

Docker Hub: https://hub.docker.com/layers/airbyte/source-mysql/3.53.2-preview.59c53ff

Registry JSON:

• OSS Registry
• Cloud Registry

[Yarden Carmeli (yardencarmeli)]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (ba041a8)

[github-actions]

Note

Autopilot progressive rollouts are not enabled for the following modified connector(s):

• source-mysql

This is a courtesy heads-up only — it does not block merge or fail any check.
To enable automatic progressive rollouts for the connector(s) above, comment
/enable-autopilot-rollouts on this PR. This sets defaultRolloutMode: autopilot
and enableProgressiveRollout: true in each connector's metadata.yaml,
preserving any existing autopilotConfig.

[github-actions]

Deploy preview for airbyte-docs ready!

Project:	airbyte-docs
Status:	✅  Deploy successful!
Preview URL:	https://airbyte-docs-uce4xeubd-airbyte-growth.vercel.app
Latest Commit:	ba041a8

Deployed with vercel-action

[github-actions]

source-mysql Connector Test Results

 10 files   10 suites   44s ⏱️
106 tests 106 ✅ 0 💤 0 ❌
230 runs  230 ✅ 0 💤 0 ❌

Results for commit ba041a8.

♻️ This comment has been updated with latest results.

[Airbyte Support Bot (airbyte-support-bot)]

↪️ Triggering /ai-prove-fix per Hands-Free AI Triage Project triage next step.

Reason: Draft PR whose only red checks are the progressive-rollout gate (unrelated to the collation fix); validating the MySQL unicode interpolation change against live connections is the next pipeline gate.

Devin session

[octavia-bot]

🔍 AI Prove Fix session starting... Running readiness checks and testing against customer connections. View playbook

• 🔗 Session URL: https://app.devin.ai/sessions/56ee64e657df467ca9a49a552011f1b9

✅ Devin AI session created successfully!

[Airbyte Support Bot (airbyte-support-bot)]

🟢 Fix Proven

airbyte/source-mysql:3.53.2-preview.59c53ff eliminates MySQL/MariaDB error 1267 (Illegal mix of collations) when partitioning a text primary key on a legacy-charset column, and the previous version reproducibly fails on the same data. Recommend review and merge.

• Controlled reproduction: on a throwaway mariadb:10.6 table whose PK is CHAR(36) CHARACTER SET utf8mb3 COLLATE utf8_general_ci inside an otherwise utf8mb4 table, populated with the reported key distribution (mostly short numeric-looking ASCII keys plus a few 36-char UUID keys, 700,008 rows, large enough to force 3 partitions): control 3.53.1 aborts on partition 3 with Illegal mix of collations (utf8mb3_general_ci,IMPLICIT) and (utf8mb4_general_ci,COERCIBLE) for operation '<=' (Error code: 1267) after 1 record; the pre-release completes with exit 0, 700,009 RECORD messages across the same 3 partitions and zero occurrences of the collation error.
• Live confirmation: the reporting team's dedicated non-production test source — already pinned by them to this pre-release, pointing at the same replica as the failing production connection — completed its first full sync of the affected stream successfully (≈41 min, first attempt, no failure summary). No pinning action was taken by Airbyte, so no approval was needed and nothing needs unpinning on our side.
• Unit evidence: at PR head ba041a8, ./gradlew :airbyte-integrations:connectors:source-mysql:test --tests '*StringInterpolationUtilsTest*' passes (38 cases), including the new assertions that no generated boundary exceeds the observed code-point ceiling and none contains an unpaired surrogate.

Next steps

1. Review and merge this PR, then publish 3.53.2.
2. Ask the reporting team to unpin the pre-release from their test source and re-enable the affected production connection on the published version.
3. Optional /ai-canary-prerelease before merge — not required here, since we have a direct control-fails / target-succeeds reproduction.
4. Two regression-harness bugs surfaced during this run are unrelated to this PR and worth filing separately: (a) the source-mysql GSM integration-test secret has an unusable SSH tunnel (SshException: No more authentication methods available), so the default credential-less regression path cannot READ at all; (b) the harness writes Cloud-fetched configured catalogs with explicit nulls for nullable fields (streams.N.stream.namespace: Null value is not allowed. (code: 1021), likewise is_resumable, default_cursor_field, generation_id, sync_id, include_files, minimum_generation_id, destination_object_name), which the Bulk CDK's strict parser rejects — this blocks READ for every source-mysql customer-connection regression run.

Details: connector, versions, baseline

• Connector: source-mysql · PR: fix(source-mysql): derive unicode interpolation base from sampled boundaries to fix collation error 1267 #83239 · Fix version: 3.53.2
• Pre-release under test: airbyte/source-mysql:3.53.2-preview.59c53ff (publish run)
• Comparison baseline (control): override_control_image=airbyte/source-mysql:3.53.1 — known-bad. The report names 3.53.0 as the first failing version and a second report names 3.53.1; 3.53.1 is both the PR's base and the latest published tag, so pinning it keeps the diff to this PR alone while still comparing against a version that exhibits the bug. No major-version boundary crossed.
• Detailed evidence (private, includes connection/workspace IDs and Cloud links): oncall#13104 comment

Evidence plan (proving / disproving criteria)

Proving: synthetic partition boundaries stay within the code-point range of the sampled min/max PK values, so a text PK on a legacy charset (utf8/utf8mb3, latin1) no longer triggers error 1267/3988; a read that previously aborted with that error completes, with stream schemas unchanged and no record loss versus control.

Disproving: the 1267/3988 error persists on the target; new errors or missing streams on the target only; or fewer records on the target than control (which would indicate lost partitions rather than a fix).

Strategy: (1) regression tests in comparison mode, all streams, cold read — required first for sources; (2) escalate to a local control-vs-target reproduction once the harness proved unable to produce READ evidence; (3) corroborate with the live sync on the already-pinned test source; (4) unit tests at PR head.

Result of the ranked attempts (all comparison mode, all streams, cold read, control 3.53.1):

1. GSM integration secrets — inconclusive: CHECK/DISCOVER failed on both versions (SshException: No more authentication methods available), READ then aborted (Catalog is required for read command).
2. Customer connection A — inconclusive: both versions failed with Timed out while opening a SSH Tunnel (DB unreachable from CI).
3. Customer connection B — SPEC, CHECK and DISCOVER green and identical on both versions (useful no-regression signal); READ failed on both due to the null-in-catalog CDK parse bug described above.
4. Customer connection C — inconclusive: both versions failed CHECK with 08S01 Communications link failure.

No attempt produced any signal against the PR: every failure was reproduced identically by the control version, i.e. it is environmental or harness-related.

Pre-flight checks

• Viability ✅ the interpolation base is derived from the highest code point in the sampled bounds (floor 0x7F), so synthetic boundaries can no longer exceed the charset actually present in the column — this is exactly the mechanism that produced the failing comparison.
• Safety ✅ boundary arithmetic plus one log line; no new I/O, credentials or network calls, no obfuscation.
• Breaking change ✅ none — patch bump 3.53.1 → 3.53.2; no spec, schema, stream, primary-key, cursor or state-format change; no user action or migration required.
• Reversibility ✅ downgrade-safe — nothing persisted changes format, so 3.53.1 can read state and config written by 3.53.2; changelog entry present.
• Design note for reviewers ⚠️ when the bounds reach the UTF-16 surrogate range (>= U+D800), interpolation is skipped and only the original bounds are returned, so a supplementary-plane key range reads as a single partition. Correct, but it is a throughput trade-off worth confirming as intended.
• Minor observation the target emitted 700,009 RECORDs for 700,008 distinct keys — one key re-emitted at an inclusive partition boundary. Partition bounds are inclusive on both sides, destinations dedupe on PK, and the control aborted before it could be measured, so this is not attributable to this PR; noted only for the record.

---

Devin session

[Airbyte Support Bot (airbyte-support-bot)]

↪️ Triggering /ai-review per Hands-Free AI Triage Project triage next step.

Reason: /ai-prove-fix returned 🟢 Fix Proven on the current HEAD ba041a83 (error 1267 reproduced on 3.53.1 and gone on the pre-release), so the PR has been marked ready for review and the review gate is the next step.

---

Devin session

[octavia-bot]

AI PR Review starting...

Reviewing PR for connector safety and quality.
View playbook

• 🔗 Session URL: https://app.devin.ai/sessions/2a6cfdf8de4e4ff8865500025c3daa93

✅ Devin AI session created successfully!

[Airbyte Support Bot (airbyte-support-bot)]

🛡️ AI PR Review Report

🟢 Review Action: APPROVED

All 12 gates passed.

🔶 Risk Level: 3/5

Logic change to existing partition-boundary generation for text primary keys in source-mysql; covered by new unit tests and a 🟢 /ai-prove-fix result.

---

📋 PR Details

Connector(s): source-mysql
PR: #83239
HEAD SHA: ba041a8366417f691c19eabd39433755b5dbda40
Session: https://app.devin.ai/sessions/2a6cfdf8de4e4ff8865500025c3daa93

🔍 Gate Evaluation Details

Gate-by-Gate Analysis

Gate	Status	Enforced?	Details
PR Hygiene	PASS	Yes	Non-empty description (1879 raw chars, ~1600 visible), changelog row added to docs/integrations/sources/mysql.md, no unresolved human review comments.
Code Hygiene	PASS	WARNING	Source change in StringInterpolationUtils.kt accompanied by test changes in StringInterpolationUtilsTest.kt.
Test Coverage	PASS	Yes	Bug fix with 5 new @Test functions plus a new assertWithinObservedCharset assertion helper.
Code Security	PASS	Yes	No changed path matches auth/credential/secret/token patterns; metadata.yaml diff hunk only changes dockerImageTag.
Per-Record Performance	PASS	WARNING	Changed code runs once per partition-boundary calculation, not per record. Minor nit: KotlinLogging.logger {} is instantiated inside unicodeInterpolatedStrings() rather than at file/object scope.
Breaking Dependencies	PASS	WARNING	No dependency or CDK version changes; Gradle Dependency Diff / Bulk CDK version check are green.
Backwards Compatibility	PASS	Yes (enforced)	No spec*.json/spec*.yaml, schemas/*.json, stream, primary-key, cursor, or type/format changes. metadata.yaml change is a patch bump 3.53.1 → 3.53.2, so no releases.breakingChanges entry is required.
Forwards Compatibility	PASS	Warning (elevates Risk Level)	No state/cursor/checkpoint/pagination/partition_router keywords in the functional diff. Partition boundaries are recomputed from sampled PK values at the start of each sync (MySqlSourceJdbcPartitionFactory.internalCalculateBoundaries) and are not part of the persisted state key format, so rollback to 3.53.1 remains safe.
Behavioral Changes	PASS	Warning (elevates Risk Level)	No rate-limit/retry/backoff/timeout/error-handler keywords in the diff hunks. Noted for reviewers: when the observed code-point ceiling reaches U+D800, interpolation is skipped and only the original bounds are returned, which reduces the number of read partitions (slower but correct) for supplementary-plane text PKs.
Out-of-Scope Changes	PASS	Skip	All changes are under airbyte-integrations/connectors/source-mysql/** and docs/integrations/sources/mysql.md.
CI Checks	PASS	Yes	38 passed / 0 failed / 0 pending on ba041a83, including Test source-mysql Connector, Lint source-mysql Connector, Build and Verify Artifacts (source-mysql), and Connector CI Checks Summary; connector test results show 106 tests, 0 failures.
Live / E2E Tests	PASS	Yes	Sync-behavior change → validation required. /ai-prove-fix posted 🟢 Fix Proven on this HEAD (comment id 5130314704): control 3.53.1 reproducibly aborts with error 1267 on a utf8mb3 CHAR(36) PK, while 3.53.2-preview.59c53ff completes 700,009 records across the same 3 partitions; a customer test source already pinned to the pre-release completed a full sync of the affected stream.

Files that triggered gate evaluation

• airbyte-integrations/connectors/source-mysql/src/main/kotlin/io/airbyte/integrations/source/mysql/StringInterpolationUtils.kt — functional change: interpolation base derived from the max code point in the bounds (floor 0x7F), plus a surrogate guard at >= 0xD800. Triggers Test Coverage and Live / E2E (sync-behavior) evaluation.
• airbyte-integrations/connectors/source-mysql/src/test/kotlin/.../StringInterpolationUtilsTest.kt — new tests satisfying Test Coverage and Code Hygiene.
• airbyte-integrations/connectors/source-mysql/metadata.yaml — dockerImageTag: 3.53.1 → 3.53.2 (patch bump only).
• docs/integrations/sources/mysql.md — changelog row for 3.53.2.

Keywords / patterns matched

• Behavioral change indicators (Test Coverage trigger): PR title contains fix; PR body contains Fixes https://github.com/airbytehq/oncall/issues/13104.
• Live / E2E category 4 (sync behavior): partition-boundary generation consumed by MySqlSourceJdbcPartitionFactory.
• No security, dependency, spec/schema, state, or operational-risk keywords matched in the +/- diff lines.

📚 Evidence Consulted

Evidence

• Changed files: 4 files (+77 / −6)
• CI checks: 38 passed, 0 failed, 0 pending, 13 skipped — Connector CI Checks Summary ✅, Test source-mysql Connector ✅, Lint source-mysql Connector ✅, source-mysql Pre-Release Checks ✅, source-mysql Progressive Rollout Gate ✅, Check Changelog Updated ✅, Format Check ✅, Docs / Vale + Docs / MarkDownLint ✅
• PR labels: connectors/source/mysql, hyd-prove, hyd-review
• PR description: present (1879 chars)
• /ai-prove-fix: 🟢 Fix Proven on ba041a83
• Human review comments: none (no reviews submitted; only bot and author comments)
• Existing bot reviews: none for this HEAD SHA
• Stage label hyd-review applied to this PR and to the originating issue airbytehq/oncall#13104

---

Devin session

[Airbyte Support Bot (airbyte-support-bot)]

↪️ Triggering /ai-ready per Hands-Free AI Triage Project triage next step.

Reason: /ai-review returned APPROVE for the exact current HEAD ba041a83 with all 12 gates PASS, /ai-prove-fix is 🟢 Fix Proven on the same HEAD, and the only red checks are the progressive-rollout gates (source-mysql has no active rollout). Running the merge-readiness evaluation.

---

Devin session

[octavia-bot-hoard]

Auto-merge evaluation: FAIL

Eligibility (all must pass)

Pass	Check	Reasoning
❌	Airbyte Support Bot Assigned	Airbyte Support Bot is not assigned; human approval and merge are required.
✅	No Destination Changes	No destination-related changes detected.

Scope analysis was skipped because deterministic auto-merge eligibility did not pass.

Workflow run

[Airbyte Support Bot (airbyte-support-bot)]

🙋 Escalated to #human-in-the-loop per Hands-Free AI Triage Project triage next step.

Routed to the DB/DW connectors oncall. /ai-prove-fix is 🟢 Fix Proven and /ai-review is APPROVE on the exact current HEAD ba041a83 (all 12 gates PASS), but the /ai-ready evaluation failed on the deterministic eligibility rule Airbyte Support Bot Assigned, so human approval and merge are the remaining steps.

---

Devin session