You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Oct 11, 2025. It is now read-only.
分布式爬取中，worker的主备mq同步问题，目前缺少主备同步机制，也就是说在主正常运行过程中，备一直接收url或者bundle放在mq中，一旦主worker挂了，备又会把主之前跑过的url或者bundle执行一遍。这样会比较耗时，可以做一个定时同步机制，这样备中的数据不会有很多冗余