request help: etcd as discover

[szylmzs]

Issue description

only four types list as discover

how do i use etcd as discover

Environment

• apisix version (cmd: apisix version):
• OS (cmd: uname -a):
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
• apisix-dashboard version, if have:
• the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
• luarocks version, if the issue is about installation (cmd: luarocks --version):

[tzssangglass]

see: #4713

[milkwine]

Apisix is a great api gateway which has a official well developed dashboard.
That's why I decide to replace our nginx proxy with apisix.
Thanks this project to make life much more easier!

It's a little bit confusing, apisix using etcd as its storage which we can't not use it as service discover center.
I have seen issue #4713 which provide a way to using etcd as service discover center.
It's a little tricky but have some problems.

Problem

We can't let online servers to register its self. In #4713 way, we must have whole view of the online servers, then can recreate the upstream in apisix. If we do it this way, things should be done: A) Server Register it's own to etcd, B) Some script watch etcd to have the whole view of this service, and then flush servers of this service to upstream in apisix.

I think B) is a much more common problem worthy to be solved by apisix.

Why consist using Etcd?

Apisix rely on Etcd, but we don't want more softwares which can cause addtional maintain work.

Temporary Solution

Coredns can response SRV query which using etcd as its backend.
So i turn to using DNS type of service discovery in apisix.
Something like: Online Server -> Etcd -> Coredns -> Apisix.

Not sure how frequent does apisix ask coredns, I do catch a lot dns query when http request comes in.
I provide every apisix its own coredns with a short duration cach , reduceing request latency(not sure sync or async), preventing from quering too much etcd when high load comes in.

Coredns: https://coredns.io/plugins/etcd/

Idea

1. Using format like SRV record in Coredns.
2. Setetcd prefix in service_name in upstream
3. Do some dns type discovery jobs and a little cache

Dose this gonna work?

[spacewander]

consul_kv is a different story.

1. consul_kv is contributed by the community.
2. We don't have Consul support before consul_kv, but we already have etcd support. So I am not so interested in the idea of etcd_kv.

[spacewander]

If TTL is desired, we can add a expire field in the node, and update it with PATCH (from Admin API) or via etcd directly.

Personally, I prefer to improve the current mechanism.

[spacewander]

Err. It doesn't work.

To use node-level TTL, we need to store node as key.
So indeed a etcd_kv service discovery is desirable.

We can accept a design purposal and PR for this.

[milkwine]

Are there any best practices about using ETCD as the discovery center? Such as the key layout, TTL settings and so on. I think using ETCD KV is a good idea but we should design a proposal firstly.

As I know coredns doing it this way(https://coredns.io/plugins/etcd/#srv-record):

% etcdctl put /skydns/local/skydns/x5 '{"host":"skydns-local.server","ttl":60,"priority":10,"port":8080}'

host gonna replaced by server's IP.

Format Compatible with Coredns

Pros is obviously, you can have your SRV record at the same time server register itself to apisix(etcd).
Then other tools support SRV are all available, such as: Prometheus's dns_sd_config.

Why not just using DNS type service discover

The registration backed by DNS will not take effect immediately during TTL caching.
Pulling from DNS server is less efficient while apisix can watch etcd natively.

[milkwine]

@milkwine Is there a standard for the etcd discovery? If the schema is custom by your company, we should not put it into APISIX. But you can write your own discovery.

Take it easy dude. Standard will not come while everybody do nothing, right?