Skip to content

MySQL: Improve parallel snapshot#3002

Merged
Amogh-Bharadwaj merged 20 commits into
mainfrom
mysql-improve-partitioning
Jun 6, 2025
Merged

MySQL: Improve parallel snapshot#3002
Amogh-Bharadwaj merged 20 commits into
mainfrom
mysql-improve-partitioning

Conversation

@Amogh-Bharadwaj

@Amogh-Bharadwaj Amogh-Bharadwaj commented Jun 3, 2025

Copy link
Copy Markdown
Contributor
  1. Fix comparison of previous partition with current partition.
  2. Use a different query to obtain logical partitions for MySQL
  • Adapt query to be able to handle timestamps

@Amogh-Bharadwaj Amogh-Bharadwaj force-pushed the mysql-improve-partitioning branch from ecc560d to a9c56b8 Compare June 3, 2025 12:19
Comment thread flow/connectors/utils/partition.go Outdated
Comment thread flow/connectors/mysql/qrep.go Outdated
Comment thread flow/connectors/mysql/qrep.go Outdated
Comment thread flow/connectors/mysql/qrep.go Outdated
Comment thread flow/connectors/mysql/qrep.go Outdated
Comment thread flow/connectors/mysql/qrep.go Outdated
Comment thread flow/connectors/mysql/qrep.go Outdated
Comment thread flow/connectors/mysql/qrep.go Outdated
@Amogh-Bharadwaj Amogh-Bharadwaj force-pushed the mysql-improve-partitioning branch from b9fc8ab to 048a692 Compare June 3, 2025 20:15
Comment thread flow/connectors/utils/partition.go Outdated
Comment thread flow/connectors/utils/partition.go Outdated
@iskakaushik

Copy link
Copy Markdown
Contributor

I think we should be able to unit test this better. One way to do it would be to create two helper structs:

PostgresPartitionedQueryGenerator and MySql.... feed them a column name and data type and assert on the generated query.

@heavycrystal

Copy link
Copy Markdown
Contributor

are there benchmarks for how much of a speedup this led to? would like to see those attached

@Amogh-Bharadwaj Amogh-Bharadwaj force-pushed the mysql-improve-partitioning branch from 4e2eeb2 to 570f979 Compare June 5, 2025 14:41
Comment thread flow/e2e/api/api_test.go Outdated
@Amogh-Bharadwaj Amogh-Bharadwaj force-pushed the mysql-improve-partitioning branch from 85b9b66 to b3272fc Compare June 6, 2025 08:42
@Amogh-Bharadwaj Amogh-Bharadwaj force-pushed the mysql-improve-partitioning branch from 082fabc to fd3d611 Compare June 6, 2025 16:48
@Amogh-Bharadwaj Amogh-Bharadwaj merged commit 5a856db into main Jun 6, 2025
9 checks passed
@Amogh-Bharadwaj Amogh-Bharadwaj deleted the mysql-improve-partitioning branch June 6, 2025 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants