Envoy rejects all requests with 503 on aggregate cluster CDS change

[codyl-stripe]

Title: Envoy rejects all requests with 503 on aggregate cluster CDS change

Description:

We’re using Envoy 1.13.0.1 with an aggregate cluster, and multiple “child” clusters.
Our aggregate cluster has a basic config which we initially supply using CDS, similar to the example given in the current docs [0].

name: aggregate_cluster
connect_timeout: 0.25s
lb_policy: CLUSTER_PROVIDED
cluster_type:
  name: envoy.clusters.aggregate
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.clusters.aggregate.v3.ClusterConfig
    clusters:
    - a
    - b
    - c

We sometimes want to change the priority levels of the aggregate cluster such as when we want to reduce the priority of cluster “a” to be the lowest. We make this change using CDS and expect the following configuration to be applied by envoy:

name: aggregate_cluster
connect_timeout: 0.25s
lb_policy: CLUSTER_PROVIDED
cluster_type:
  name: envoy.clusters.aggregate
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.clusters.aggregate.v3.ClusterConfig
    clusters:
    - b
    - c
    - a

What we actually experience is envoy will accept this change from the original aggregate cluster to the new one, but will start returning 503s for all requests to the aggregate cluster.

curl -v -X POST http://localhost:17777/check
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 17777 (#0)
> POST /ping HTTP/1.1
> Host: localhost:17777
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 19
< content-type: text/plain
< date: Thu, 04 Jun 2020 22:55:38 GMT
< server: envoy
<
* Connection #0 to host localhost left intact
no healthy upstream

Checking the relevant child cluster stats (with curl http://localhost:9901/clusters | grep health_flags), we see that all child clusters are still healthy and even enabling health check debug logs we see that the child clusters continue to healthcheck healthy:

[2020-06-04 23:01:42.026780] [2020-06-04 23:01:42.026][148123][debug][hc] [external/envoy/source/common/upstream/health_checker_impl.cc:264] [C28240] hc response=200 health_flags=healthy
[2020-06-04 23:01:42.048145] [2020-06-04 23:01:42.048][148123][debug][hc] [external/envoy/source/common/upstream/health_checker_impl.cc:264] [C28234] hc response=200 health_flags=healthy
[2020-06-04 23:01:42.059999] [2020-06-04 23:01:42.059][148123][debug][hc] [external/envoy/source/common/upstream/health_checker_impl.cc:264] [C28233] hc response=200 health_flags=healthy
[2020-06-04 23:01:42.064204] [2020-06-04 23:01:42.064][148123][debug][hc] [external/envoy/source/common/upstream/health_checker_impl.cc:264] [C28237] hc response=200 health_flags=healthy
[2020-06-04 23:01:42.071922] [2020-06-04 23:01:42.071][148123][debug][hc] [external/envoy/source/common/upstream/health_checker_impl.cc:264] [C28238] hc response=200 health_flags=healthy
[2020-06-04 23:01:42.151415] [2020-06-04 23:01:42.151][148123][debug][hc] [external/envoy/source/common/upstream/health_checker_impl.cc:264] [C28247] hc response=200 health_flags=healthy
[2020-06-04 23:01:42.157455] [2020-06-04 23:01:42.157][148123][debug][hc] [external/envoy/source/common/upstream/health_checker_impl.cc:264] [C28248] hc response=200 health_flags=healthy

So we know that all child clusters still have healthy hosts in them, but envoy continues to assert that there are no healthy hosts when we use the aggregate cluster and returns 503s.
Similarly, the connection pools continue to report that they contain no healthy hosts. Enabling upstream debug logging (with curl -X POST http://localhost:9901/logging?upstream=debug) produces:

[2020-06-04 22:58:47.102169] [2020-06-04 22:58:47.102][148145][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1230] no healthy host for HTTP connection pool
[2020-06-04 22:58:47.302183] [2020-06-04 22:58:47.302][148145][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1230] no healthy host for HTTP connection pool
[2020-06-04 22:58:47.502154] [2020-06-04 22:58:47.502][148145][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1230] no healthy host for HTTP connection pool

Restoring the aggregate cluster to its previous state via another CDS update does not rectify the situation, and we have found that the only way to get envoy out of this state of always returning 503s is to restart the instance.

Relevant Links:
[0] https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/aggregate_cluster#example
Thanks!

[mattklein123]

Seems like a warming bug or just a straight up bug. cc @snowp.

[snowp]

Definitely sounds like a bug, I'll try to reproduce this in an integration test.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[VassoSof]

Hello, I would like to revive this thread. We are experiencing this issue with 14.1. Is there any analysis outcome. @snowp is this case reproduced? Possibly solved with 1.15?

@mattklein123 as to be notified

Thank you!

[snowp]

Hey I haven't had the time to look into this, I'll see if I can find some time this week.

cc @yxue in case you have any idea what might be up

[VassoSof]

@snowp Did you have the time to check this issue? There are more people reporting this problem. Aggregate cluster is malfunctioning!

[snowp]

At this point I have a reproducing test case, and it seems like the issue is that when the aggregate cluster is loaded via the active map (ie not during initial load) the call to populate the TLS clusters happens before the cluster has been initialized, so the TLS cluster doesn't yet exist.

I'm not sure yet what the solution is here, posting the call to refresh on the dispatcher fixes the ordering issue but introduces a window in which the cluster has finished warming but isn't ready to receive traffic since the computed lb structures haven't been loaded. Tying this into the init manager doesn't seem to work either since that seems to prevent the cluster from initializing at all, so we never get to the point where we can initialize the TLS clusters.

It might make sense to introduce some kind of initialization phase for the TLS cluster - that way the new TLS clusters would call back into the main cluster to be updated with whatever they need. Will require some thinking

[VassoSof]

@snowp Thank you for your detailed feedback. Waiting for your decision to correct. Could we have the solution on top of 1.14.4?

[VassoSof]

@snowp Might you share a date by when you could communicate the decision?

Regards,
Vasso