Add service discover for -mesh.peers

[iksaif]

Currently -mesh.peers takes a static list of host:port. If you want to use service discovery to discover your peers you need to do that before the command line invocation.

It more or less work with consul because you can use something like dig +noall +answer foo.service.dc.consul SRV, but that is far from optimal.

Do you think it would make sense to add service discovery support for this in the alertmanager ?

[stuartnelson3]

I think this sounds like a good addition, but it is worth noting that the current gossip model will discover additional peers. The list you provide at the start is not the "set in stone" list of your peers. For example, if you start peerA, and then start the following peers B, C, and D, all configured with peerA as their initial peer list, they will discover each other eventually.

As far as finding an additional list of peers, though, I do think supporting service discovery makes sense.

[brancz]

I've spoken to @fabxc about this and we also came up with something in the direction of what @stuartnelson3 wrote. It is mainly interesting for the initial discovery, and for that reason we didn't think taking on the huge dependency, would be worth it. Actually dynamically discovering and changing peers could disrupt the mesh network, so it may actually harm it - or seem like it doesn't work because it would in fact not dynamically reconfigure like Prometheus does.

The way we handle the mesh network initialization it in the Prometheus Operator is that we don't connect the Alertmanagers through an IP address but through a stable DNS name. All instances deployed then have all other instances as arguments. While this may look odd when scaling up and down it actually always works out as all initial instances are always part of the mesh and by the mesh sharing all connections all peers are eventually connected to each other.

[matthiasr]

Since it's only used at startup, I think relying on scripting around for cases where DNS is not available is good enough, compared to the complexity of having full blown SD but then using it very differently. Keep in mind that many Prometheus SDs (all but DNS and file?) rely on relabeling to actually select targets from the global lost, so we'd have to pull that in too.

I would reconsider if someone can demonstrate a case that is trivial with SD and impossible to solve otherwise, but for now I'd say no.

[brian-brazil]

I think we'll ultimately have to add it, but let's not do it earlier than we need to.

[fabxc]

I don't think we ever have to add it. There is really no point as it is only needed during bootstrapping.

Agreeing with @matthiasr on all ends.

There any script that populates the startup flags with results from querying service discovery serves the purpose just fine. All the complexity of relabeling makes the cost/benefit of adding this underwhelming.
Being not a stateless application and having a reachable UI all instances of AM will likely have a stable DNS name and in general not even a complex script is required querying an actual SD system.

I'd like to close this as I consider it a non-goal for the time being and don't want long-standing "maybe but probably not" issues idling in the tracker.

[iksaif]

I guess the idea was that it would be more user friendly not to have to script it, but it certainly is not a priority

[fabxc]

I could see it as part of amtool #636 possibly but it's probably not trivial to do without still having a config file for it.

[stuartnelson3]

Closing issue as it is currently not planned to be implemented

[discordianfish]

Could we at least support DNS RR based discovery for the initial peers? That seems straight forward and it's what consul is doing too for example. Basically just resolve the DNS name and use all returned IPs as initial peer list. Without that, you need a rather complex kubernetes setup to still make it work. And even in that case, you need to generate the manifest because 'replicas' need to match the parameters.
With "DNS SD" all you need is a deployment and headless service.

[fabxc]

I think DNS is reasonable request and makes a lot of deployments significantly simpler. I'd only agree to this though if we can make a firm decision to leave it at that. I've no interest in supporting the full set of SD we do in Prometheus. Especially as all the meta data non-DNS gives us is not required here.

…

On Wed, Jun 28, 2017 at 7:20 PM discordianfish ***@***.***> wrote: Could we at least support DNS RR based discovery for the initial peers? That seems straight forward and it's what consul is doing too for example. Basically just resolve the DNS name and use all returned IPs as initial peer list. Without that, you need a rather complex kubernetes setup to still make it work. And even in that case, you need to generate the manifest because 'replicas' need to match the parameters. With "DNS SD" all you need is a deployment and headless service. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#645 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEuA8vHTrA0tCXd7cFcO-tPFOkQ3i61rks5sIotsgaJpZM4MVkcW> .

[iksaif]

Maybe full DNS-SD support isn't required and just supporting SRV records (to be sure to get the right port, which is important if you run in marathon/kube/whatever).

Currently we need to do that:

#!/bin/bash                                                                                                                                                                                                       
                                                                                                                                                                                                                  
set -x                                                                                                                                                                                                            
service=$1                                                                                                                                                                                                        
peers=                                                                                                                                                                                                            
for dc in dc1 dc2 dc3 dc4 dc5 dc6; do                                                                                                                                                                     
  peer=$(dig +noall +answer ${service}.service.${dc}.consul SRV  | perl -pe 's/.+\s+(\d+)\s+(.+)\.$/-mesh.peer=$2:$1/g')                                                                                          
  if [[ -n $peer ]]; then                                                                                                                                                                                         
     peers="$peers $peer"                                                                                                                                                                                         
  fi                                                                                                                                                                                                              
done                                                                                                                                                                                                              
if [[ -z $peers ]]; then                                                                                                                                                                                          
    export ALERTMANAGER_MESH_PEERS_OPTION=                                                                                                                                                                        
else                                                                                                                                                                                                              
    export ALERTMANAGER_MESH_PEERS_OPTION=$peers                                                                                                                                                                  
fi                                                                                                                                                                                                                
echo $ALERTMANAGER_MESH_PEERS_OPTION 

And alertmanager -mesh.peers $(./mesh_peers.sh) which is hacky :)