Support retaining clusters when still referenced by route/listeners

[howardjohn]

Title: Support retaining clusters when still referenced by route/listeners

Description:

Currently, our ingress's gets clusters for all services we know about. This is often extremely wasteful, since most of them are not referenced by any routes.

We added an option to filter these out, but started seeing some issues when

• Initial state: route /foo -> cluster foo-v1
• Desired end state: route /foo -> cluster foo-v2, remove foo-v1

In practice, there is a gap between the cluster being removed and the route being updated. This results in a period of downtime on route updates.

Our ideal would be that envoy doesn't delete the cluster while a route still references it.

---

The alternative would be to solve it in the control plane. But that seems pretty hard... I guess we would need to realize foo-v1 is referenced by a route and keep it around until the route is updated.

This may be ~impossible for us (until we move to Delta XDS, anyways), since "keep it around" with sotw means we literally need a copy of it in the control plane and we don't store config after we send it, nor can we generate it again.

And to know its updated we need to send the route and wait for the ack I guess? Which also introduces a ton of complexity to the control plane and requires it to be stateful.

It seems a bit complex, and I am not even sure it can work if the proxy reconnects to a new control plane instance during the middle of this operation, unless our control plane instances sync data which is extremely complicated

[optional Relevant Links:]
Issue tracker on our side: istio/istio#29131

[Shikugawa]

Our expectation is that prevent the immediate drain of the cluster which is bound to the route when the route is updated via RDS, and wait to update the cluster until the next cluster is available. Is it correct?

[howardjohn]

Yeah that would be ideal I think.

[mattklein123]

Adding this is going to be very complicated and IMHO this is a control plane problem concern, not an Envoy concern. I'm going to push back pretty hard on adding this unless there is a very good reason.

[howardjohn]

I actually generally agree with that, but my concern is that I am not convinced a control plane can actually solve this in any reasonable way.

Consider:

1. Initial state: route /foo -> cluster foo-v1. Connecting to CP-1 instance of control plane.
2. User config changes to point /foo -> cluster foo-v2.
3. Control plane is smart, so it sends the new cluster foo-v2 first.
4. Control plane crashes, new instance CP-2 starts up and envoy reconnects
5. Now CP-2 needs to resend all config. It seems that the user wants /foo -> cluster foo-v2; no config needs cluster foo-v1 (and in fact, CP-2 may not even know foo-v1 is something that even exists at all!), so it removes foo-v1.
6. Traffic is now broken
7. Eventually CP-2 finishes the push, and moves /foo -> foo-v2. Traffic is now fixed

In our specific case, our control plane is essentially always in the 'Control plane crashes' state, because we do not maintain the state required to handle this complex operation; I used a crash as that is something that I would assume no control plane is immune from.

[mattklein123]

IMO if your control plane crashes it's sufficient for there to be a delay until things converge, downstream retries, etc. should take care of this.

[howardjohn]

When the control plane crashes I would expect configuration updates to be delayed. I would not expect traffic to be broken (ie 503s until control plane comes back up).

[mattklein123]

When the control plane crashes I would expect configuration updates to be delayed. I would not expect traffic to be broken (ie 503s until control plane comes back up).

I don't understand why your control plane can't come up in a consistent state and send the new config.

[howardjohn]

The users state is configured in Kubernetes. This includes service discovery (Services/Endpoints), as well as routing configuration. In Kubernetes, we can only get access to the current configuration - we cannot view historical state, unless we maintain that out of band.

In step 3 from #18958 (comment), the control plane can do this because it saw the old config, and now it sees the new configuration, so it could be smart enough to know we need to "stage" the update in two parts (in theory - Istio cannot do this today).

With a new control plane coming up though, there is no way it can understand the state of the old config. So all it sees is the user has config for a foo-v2 Service (ie foo-v2 Envoy Cluster), at which point it doesn't really have much choice other than to send just foo-v2. Even if it somehow knew it should send foo-v1, we don't have the Service for foo-v1 so we wouldn't know how to synthesize a cluster for it.

Its possible I am over complicated this :-)

[mattklein123]

I understand what you are saying in terms of the corner case, but I don't think Envoy holding configuration is the right solution to this problem. It's too complicated. If you need atomic updates I would rather just figure out a way in which a route update is held until all of the clusters it references are actually available. This has been discussed in the past. cc @htuch

[htuch]

Yeah. The neat way to approach this would be to have the xDS Resource declare its dependencies in the wrapper and have the xDS machinery wait to deliver until these are available from all sources. The downsides are that this is going to add extra complexity and debug pain when using this feature (it would be opt-in), and our xDS delivery right now is not really setup as a cache (where this would be more natural). CC @adisuissa