Broker and project organisation

[prdn]

@maxired I like the idea to let devs to override the routing system.
Keeping that in mind we should think to the best solution performance-wise.
We should have a method in the broker like route that allows devs to override the default worker/service selection behaviour.

I think that we should have the tiniest broker possible and then create a separate project with addons to override default behaviours.

The wildcard and versioning system can be viewed at the moment as "core" features because they are really useful in quite any scenario. Anyway we may want to make these features as addons in the future.
For sure we have to maintain the cleanest and tinies code base possible.

The future I see for pigato broker is mainly a directory service and load balancer.
Something that I really want to have is that the broker should continue to act like a router but it should also let worker and clients speak directly if possible (depending on some factors like network access, authorisation, ....)

Let's start moving into that direction.

[maxired]

@prdn regarding additional behaviour, I wish I have priority over service.
With the current implementation, I am not sur whether a can do that easily, even with the semver mechanism.

I totally agree that we want stuff to be small and with good performance.

Regarding the future of the broker and client-to-worker communication, a solution that might work would be in a first time to remove the broker, and doing directly client-to-worker communication.
On top of this mechanism, we can create a broker which will be a specific worker, instantiating client and forwarding requests to others workers. This would involved lots of change in the low level, but not much from a user perspective.

[maxired]

@prdn , @allain, what do you think would be the best way for a service to give parameters to a custom routing function in order to implement a custom routing ?

I came up with two solution :

• Add one more semantic scheme to service name. We already have '*' for wildcard and '@' for semver. In order to allow anyone to add custom parameters, we should think of something more flexible. I was thinking about the syntax used by http query param : ?param1=value1&param2=value2
• used the opts field send with the heartbeat message. We kind of already use it for concurrecny setting, but the way it is currently implemented is not suitable for others options.

I am personnaly not a big fan of the first solution, but I think this would be simple to implement whearas the second one might have lots of side effect, since we use the service name in lot of place.
A reason that I don't like the first solution is because it can confuse people used to pattern matching, ie people using seneca.

Any thought on this ?

[allain]

The way I see custom logic being implemented is as services themselves. If you want your own tweaked logic, you modify what service is responsible for it. When we have client => worker communication going, we can increasingly reduce the load on the Broker this way.

For example, when customizing service resolution we specify it as so:

new Broker('tcp://...', { conf: {
  'resolver': 'custom-service-name' 
}});

// or for the simplest case
new Broker('tcp://...', { conf: {
  'resolver': worker // worker which implements the 'resolver' service
}});

If a request comes in for a service that isn't known, it passes it on to the resolver service, which returns a list of service names in whatever order it deems best. (possibly most advanced semver, or * match witch has the least matching services in it), geolocation, etc).

Clients can then perform resolution by directly querying the resolver service, leaving the Broker out of it except for the initial hookup.

[prdn]

I like these ideas.
The resolver worker that handles any non default routing is a good solution in my mind.
Our default resolver will offer wildcard, server and parametric routing.

[allain]

Personally I'm pushing for workers all the way down with a smaller core since if the model is good enough for users, it should be good enough for us internally. It'd force us to tightening up the core loop (not that it's bad atm), and come up with some optimizations to it. (Think same process message queue bypassing, etc).

If this idea has merit to you, I can start on a branch that refactors some of the core out in this way. If clarity goes up and performance doesn't dropp too greatly, then I think it's a win.

Dog Fooding FTW

[prdn]

@allain the all workers solution along with a smaller broker is the solution, so let's move on on that path.
If you can help with refactoring it would be great, thanks