Sending Message to Bus fails if called Too Soon

[techman83]

Credit to @pjf as his brain wave figured it out. Code to duplicate the issue:

#!/usr/bin/env perl
use Exobrain;

my $exobrain = Exobrain->new;

$exobrain->notify("Hello World");
sleep(1);
$exobrain->notify("Hello World 2");

Results in only "Hello World 2" making it to the notification service.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[pjf]

Merging from #54:

---

From #52, there appears to be a condition where we can try to send things across the bus before it's fully connected. This is definitely something which can happen with ZMQ, and may be seen with something like:

use Exobrain;
my $exobrain = Exobrain->new;
$exobrain->notify("Hello World 1");
sleep(1);
$exobrain->notify("Hello World 2");

Solutions could be:

• Provide a $exobrain->connect or equivalent message to make sure we're actually connected to the bus. We'd need to say if we're going to be subscribing or publishing; so it may make sense to just call $exobrain->sub or $exobrain->pub` (which already exist, and will initiate the connection).
• Actually be able to mark packets as important (eg, with intents), or configure a basic buffer that needs to fill before they get dropped (which is probably sufficient). I know that ZMQ has a way of specifying buffer sizes, although I have no idea if they exist on PUB sockets. This could just be a Simple Matter Of Code.

This ticket may also be a duplicate of #52 if this is the actual cause. (It may not be).

Many thanks to @techman83 in helping find this.

[techman83]

Looking at the doco, this may actually be expected behavior!

There is one more important thing to know about PUB-SUB sockets: you do not know precisely when a subscriber starts to get messages. Even if you start a subscriber, wait a while, and then start the publisher, the subscriber will always miss the first messages that the publisher sends. This is because as the subscriber connects to the publisher (something that takes a small but non-zero time), the publisher may already be sending messages out.

Though there may be still a small delay after the initial message.

#!/usr/bin/env perl
use feature qw(say);
use Exobrain;

my $exobrain = Exobrain->new;
my $count = 0;

while ($count < 11) {
  say "Hello World $count";
  $exobrain->notify("Hello World $count");
  $count++;
}

Which produces:

Hello World 0
Hello World 1
Hello World 2
Hello World 3
Hello World 4
Hello World 5
Hello World 6
Hello World 7
Hello World 8
Hello World 9
Hello World 10

However only the following hits the log + pushover:

2014/05/18 11:00:53 INFO Hello World 2 
2014/05/18 11:00:53 INFO Hello World 3 
2014/05/18 11:00:53 INFO Hello World 4 
2014/05/18 11:00:53 INFO Hello World 5 
2014/05/18 11:00:53 INFO Hello World 6 
2014/05/18 11:00:53 INFO Hello World 7 
2014/05/18 11:00:53 INFO Hello World 8 
2014/05/18 11:00:53 INFO Hello World 9 
2014/05/18 11:00:53 INFO Hello World 10 

However it's not consistent. Sometimes it only misses the first message, sometimes it's two or three messages. Though if you put a sleep after the first message it seems happy enough.

Not sure what the best solution is. The delay needs to be just long enough to allow the tcp connection to finish it's handshake. Though if we move beyond just localhost connections (something I desire), this delay could be quite unknown.

[techman83]

I've been sporadically doing quite a lot of reading about this issue and pondering ways to solve it. Overall ZMQ appears to have been designed for messaging performance, so the reality is that if the messages are important they need to have acknowledgments.

I think to begin with, the best idea will be to send a message to the bus and have a short wait as a exobrain->connect function as suggested. And in the long term add in received acknowledgements and timeouts.

[techman83]

Submitted #57 that appears to reliably fix this on localhost. However it may require some further testing over a latent link. Though Exobrain's current design is for local connections, so this should more than satisfy the requirements.

[techman83]

Had a bit of a chat with @domm regarding this issue. Setting up some sort of message acknowledgement could add a whole bunch of different things to think about, along with complications.

There are ZMQ_EVENTS that we might be able to lean on, but I'm unsure how backwards compatible that is.
http://api.zeromq.org/4-0:zmq-getsockopt#toc25

There is probably a way to automate the connect function and use a sleep for older versions, loop for newer. I'll refresh myself and take a look next time I have some spare cycles.