EventBus addresses registration in the cluster are not immediately visible by other nodes

[Igor-Polatajko]

Hello

We are using Zookeeper (from AWS MSK) as a cluster manager for Vert.x.
In our case, we rely a lot on immediate cluster consistency, while registering new addresses in the cluster, but vertx-zookeeper or Zookeeper itself seems not to provide the required consistency.

Scenario, in which we observe the issue:

1. Client node want to talk to consumer-1
2. consumer-1 address is not registered, so Client node requests consumer-factory to create a consumer for address consumer-1.
3. Request to consumer-factory happens to fall to Node 1, Node 1 processes registration request and creates consumer for address consumer-1.
4. Client node again tries to talk to consumer-1 (almost immediately, after the response from consumer-factory).
5. consumer-1 is not yet visible by Client node(!!!!!)
6. Client node again requests consumer-factory to create a consumer for address consumer-1.
7. This time request to consumer-factory happens to fall to Node 2, Node 2 processes registration request and creates yet another consumer for address consumer-1.
8. We end up with double registration of consumers for address consumer-1, which causes us a lot of bugs because our consumers are stateful.

Simplified sample of code, that we use for consumer registration:

    var consumer = vertx.eventBus().<String>consumer("address");
    consumer.completionHandler(result -> {
      // at this time, we need an address to be visible for all nodes of the cluster
    });

Presume, that currently, we cannot change the architecture and we just want the consumer, registered at step "3", to be immediately visible by all the nodes in the cluster.
Is there any way to ensure immediate consistency for the cluster (make completionHandler of consumer registration to be invoked only after the address is propagated to all nodes)?
Is this an expected behavior of the cluster manager?

vertx-zookeeper version, that we are using: 3.9.9