Events are delayed for new and (or) long running connections

[mcferden]

I have the following setup:

• Django app running in WSGI mode
• Multiple Celery workers sending events
• Pushpin (official Docker image with standard configuration, except disabled rate limit)
• Nginx in front of both Django and Pushpin
• Pushpin is only proxying requests to /events/ route

When the number of active connections to events endpoint becomes sufficiently large (~2000), all new connections start to receive events only after some time (from couple of seconds to minutes), and it's also true for already active connections which haven't received events for some time. However, Pushpin's logs show that Pushpin is receiving those events and has active subscribers.

What I've noticed is every time new connections to /events/ endpoint are created, Pushpin sends a request to backend with some JWT, and also send the same requests for each running connection with some fixed interval. And as far as I can see, subscriber will not receive any events, until such request finishes for this connection. With large enough number of connection Pushpin creates constant stream of requests which are blocking eachother, causing interruptions in event delivery.

So my question: is it expected Pushpin's behaviour, or there is some missconfiguration causing this effect?

[jkarneges]

django-eventstream configures each connection with a "next link" for eventual consistency purposes, which Pushpin will indeed fetch on an interval. Next links are unique per Django user and per set of subscribed channels. Pushpin tries to conflate requests to unique links, so for example if N connections are configured with the same link then Pushpin will only make 1 request to the link on behalf of those N connections.

My guess is you've either got a lot of unique subscription permutations or you're using Django auth, resulting in a lot of unique next links. Can you confirm this?

Are you configuring a custom channel manager? If not, and if Django auth is what's causing the uniqueness, then a quick fix could be to use some middleware hackery to prevent django-eventstream from seeing the session/user on the /events/ route since it's not needed anyway. We could also consider updating the library to be smarter about this.

[mcferden]

Thank you for clarifying the situation.

My guess is you've either got a lot of unique subscription permutations or you're using Django auth, resulting in a lot of unique next links. Can you confirm this?

Yes, exactly this. I use custom channel manager that for each client defines 3 channels:

• broadcast
• account-{account_id} (accounts link multiple users)
• user-{user_id} (regular Django user)

So, basically, there will be at least one unique channel per subscriber.

In this case the best solution for now will be to upgrade django-eventstream and migrate from Pushpin to Redis, I guess?

[jkarneges]

Ah ok, so you've got unique channels per user which will result in unique next links regardless of the use of Django auth.

Are you calling channel_permission_changed(), or using storage?

The Redis feature currently doesn't distribute permissions changes and it doesn't have a recovery mechanism. Whether that's okay will depend on what you're doing.

[mcferden]

Are you calling channel_permission_changed(), or using storage?

No, I don't use any of these features.

I've tested Redis setup and it works just fine for my use case. Regarding recovery mechanism — do you plan to add one in the foreseeable future?

[jkarneges]

Yeah you should be fine then. If you weren't using storage, then the recovery mechanism would not have been in use anyway.

We ought to make it possible to do recovery when using Redis, although I imagine there will be scaling challenges similar to the way it works with Pushpin, i.e. lots of unique channels will mean lots of DB queries. No ETA but we'll be thinking about it.