Lookup results are inconsistent between peers, and don't get lower on termination of proving node #213
Description
I am really having trouble with distributed actor swarm.
Outline
Remote Actor lookup result between peers are not consistent, and even increasing.
Here is the brief setup.
Configuration
All nodes are on the different machines
- Public peer node A, B with known Addr, and fixed identity (peer_id)
- Agent node registering a Actor
- Local node looking up the node
Observation
Log confirms
- Public peer nodes start up and add each other's id and addr, and dial, and successfully connected.
- Agent node starts up after good amount of time public peer nodes are prepared, adding peer_id and add of both bootstrap node, and dial to both of them with successful connection. After at least one successful connection, it spawns and register Actor.
- Only one of two public peer node reports result of
lookup_allis 1, another still reports 0 (INCONSISTENT!) - If I turn on local node, adding both public peer node, and dialing them with successful connection, it reports lookup result is 0.
- If I turn off and on Agent node, only one of public peer node's lookup count increases, and never get lower on termination of agent node, resulting the lookup count keep increasing. Local nodes keeps reporting 0 result.
Issue
- I thought the lookup result should be the same for connected peers but it's not.
- The lookup count should be decrease if certain node terminated (as its provider also terminated) but it's not.
So are these intended behavior? If not what would be the possible reason? Can I get working example not using mDNS and for actual remote network configuration? (including some known public peers, participating non-public nodes)