fix mysql stuck on syncer.Close()#4195
Merged
Merged
Conversation
ilidemi
reviewed
Apr 18, 2026
| // Let CDC streaming establish before blocking the tunnel. | ||
| time.Sleep(2 * time.Second) | ||
|
|
||
| // Black-hole the SSH transport so syncer.Close() hangs |
Contributor
There was a problem hiding this comment.
If SSH is healthy in this case, should toxi be declared on top of MySQL?
Contributor
Author
There was a problem hiding this comment.
that would be more direct, in this case was trying to reuse existing code for simplicity given what we are testing here is kind of trivial which is the watchdog works. maybe it's not worth a whole e2e test at all and better to test something abstract.
Contributor
There was a problem hiding this comment.
I'm not sure the abstraction is needed here, given how short it is, and it comes with two callbacks. The scenario we want to test is being able to finish the batch when MySQL connection is down, and if the infra doesn't allow it, the infra needs to be better.
Better infra at #4207
❌ 2 Tests Failed:
View the top 2 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
Contributor
🔄 Flaky Test DetectedAnalysis: TestMySQLCloseSyncerWithTimeout fails with "connection refused" on local port 42006 when setting up an SSH tunnel, a network/timing error consistent with a port conflict or race condition under high test parallelism (-p 32). ✅ Automatically retrying the workflow |
Contributor
❌ Test FailureAnalysis: New test |
Contributor
🔄 Flaky Test DetectedAnalysis: TestCreateCDCFlowAttachIdempotentAfterContinueAsNew failed with a race-condition-style assertion ("1" is not greater than "1") — the async Temporal ContinueAsNew increment had not yet occurred when the check ran, in a test suite already running close to its 900s timeout. ✅ Automatically retrying the workflow |
Contributor
🔄 Flaky Test DetectedAnalysis: The test failed due to PostgreSQL SQLSTATE 57P01 ("terminating connection due to administrator command"), an external infrastructure event that forcibly killed the catalog connection mid-test — not a code defect. ✅ Automatically retrying the workflow |
ilidemi
reviewed
Apr 18, 2026
ilidemi
reviewed
Apr 18, 2026
ilidemi
approved these changes
Apr 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When mysql connector is set up with ssh tunnel, there is a possible deadlock observed where the syncer is never closed and PullRecords hangs indefinitely.
Notice the
could not set read deadlinewarning, and thekill connectionerror, both coming from mysql binlogSyncer:could not set read deadlineis failing with error message "ssh: tcpChan: deadline not supported"kill connectionis failing with error message ""error":"ERROR 1094 (HY000): Unknown thread id: 743468""What happened here is the library we are using for processing mysql replication, go-msyql, has a bug that can manifest if it's using SSH tunnel and if the thread is already reaped by the MySQL server. This let the
syncer.Close()to hang indefinitely waiting because go-mysql thinks the connection is still alive (and the ssh tunneling connection very much is) so it gets stuck onwg.Wait()insideBinlogSyncer.close()This is a different case than #4067, because SSH tunnel itself is healthy.