全量同步数据改成协程池 切分主键进行同步

[chengcxy]

首先获取表的最小最大id,根据配置文件的bulk Size参数切分区间,创建$Maxprocs个worker
比如 mysql表最小id为1，最大为100万,bulk Size=5000,Maxprocs=20
会先创建20个worker等待执行任务,根据最小最大id生成下面的子任务
where id>0 and id<=5000
where id>5000 and id<=10000
where id>10000 and id<=15000

每个worker从taskChan取任务，执行完将任务结果写入resultChan通道 最后主进程汇总任务的结果

➜ go-mysql-transfer git:(master) ✗ go run main.go -stock true
2022-12-24 19:24:20.868626 I | process id: 32925
2022-12-24 19:24:20.868650 I | GOMAXPROCS :24
2022-12-24 19:24:20.868653 I | source mysql(127.0.0.1:3306)
2022-12-24 19:24:20.868657 I | destination rabbitmq(amqp://guest@guest@*****:5672/)
2022-12-24 19:24:20.918500 I | bulk size: 10
2022-12-24 19:24:20.918725 I | 开始导出 blog.blog
2022-12-24 19:24:20.918744 I | minSql select min(id) as min_id from blog.blog
2022-12-24 19:24:20.918754 I | maxSql select max(id) as max_id from blog.blog
2022-12-24 19:24:20.919563 I | minId:1,maxId:67
2022-12-24 19:24:20.923962 I | blog.blog 导入数据 10 条
2022-12-24 19:24:20.949198 I | blog.blog 导入数据 9 条
2022-12-24 19:24:20.955680 I | blog.blog 导入数据 9 条
2022-12-24 19:24:20.958201 I | blog.blog 导入数据 8 条
2022-12-24 19:24:20.958359 I | blog.blog 导入数据 7 条
2022-12-24 19:24:20.959305 I | blog.blog 导入数据 7 条
2022-12-24 19:24:20.959734 I | blog.blog 导入数据 10 条
2022-12-24 19:24:20.959745 I | 共耗时 ：41（毫秒）
2022-12-24 19:24:20.959749 I | 表:blog.blog 共导入 60条