Each harvester microservice should store its state and make it available to other harvesters. This way multiple harvesters can communicate with each other and work on different ranges of data. It prevents processing duplicate messages more than necessary. Such stateful information would include the current page of search results, etc.
This should be easily done using boltdb and exposed as a RPC via Kite. Should a service go offline and the same data be processed by another service (duplicate work), it should still be OK. While slightly inefficient it is "safer" and more consistent. The architecture is also setup in such a way that duplicate messages can not exist in the database.
Each harvester microservice should store its state and make it available to other harvesters. This way multiple harvesters can communicate with each other and work on different ranges of data. It prevents processing duplicate messages more than necessary. Such stateful information would include the current page of search results, etc.
This should be easily done using boltdb and exposed as a RPC via Kite. Should a service go offline and the same data be processed by another service (duplicate work), it should still be OK. While slightly inefficient it is "safer" and more consistent. The architecture is also setup in such a way that duplicate messages can not exist in the database.