Deep subscriptions

[fgalan]

Is your feature request related to a problem / use case? Please describe.

To separe csubs that has to be keep in the csubs cache (which actually is a memmap rather than a cache ;) that csubs that should be keep only in database.

This could be needed in some cases in which the CB memory doesn't suffice to keep all the csubs in memory. This is the case of a huge number of heavy subscriptions ("heavy" in the sense of lot of triggering attributes, very long JEXL conditions, etc.)

Describe the solution you'd like

Modifications in csubs and updates:

• New field in subscription, meaning that the subscription must be keep only in DB, e.g. "deep": true.
• By default updates only look for subscription in the csubs cache. However, if the update is marked somehow (e.g. ?options=deepSubscriptions) then in addition a search in DB for deep subscriptions is done.

Note that in the case of using the -noCache flag subscriptions are always searched in DB (no matter if they are marked as deep or not and/or if the update specifies the deep subscriptions option). However, -noCache is not used in production deployments.

Describe alternatives you've considered

An actual cache, but it will not work. For instance, if we keep 20% of the "hot" subscriptions in memory, there is no guarantee that a given update fall into the remaining 80% so a DB lookup will be always needed. Without active collaboration of the client sending the update (using the deepSubscription option for instance) it is very hard to solve.

Describe why you need this feature

• To improve or simplify an scenario.