aggregate cluster extension failover not working as expected

[vinzo99]

Title: aggregate cluster extension failover not working as expected

Description:

To feature IPv4/IPv6 dual stack support, Envoy's aggregate cluster functionality is used, so that the traffic can go through either the IPv4 or IPv6 networks, where the priority is defined using a cm-configurable option where we can set the preferred IP-version network.

The normal scenario works as expected, where the traffic is going through what is set to be the primary cluster in the aggregate cluster setup. However, when the primary cluster fails to connect, the secondary cluster (which should normally connect) is not being routed any traffic, and a 503 error is returned:

[debug][upstream][source/common/upstream/cluster_manager_impl.cc:1235] no healthy host for HTTP connection pool
[debug][http][source/common/http/conn_manager_impl.cc:1475] [C111][S15900032332570351549] Sending local reply with details no_healthy_upstream

Here is a sample of the aggregate cluster configuration :

{
     "version_info": "3",
     "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "UI-Nudm_NF-udm_instance_1_FQDN-udm01.udm.5gc.mnc03.mcc262.3gppnetwork.org_192.168.5.14",
      "type": "EDS",
      "eds_cluster_config": {
       "eds_config": {
        "ads": {}
       }
      },
      "connect_timeout": "5s",
      "lb_policy": "LEAST_REQUEST",
      "circuit_breakers": {
       "thresholds": [
        {
         "max_requests": 10000,
         "max_retries": 3
        },
        {
         "priority": "HIGH",
         "max_requests": 10000,
         "max_retries": 3
        }
       ]
      },
      "http2_protocol_options": {
       "hpack_table_size": 4096
      },
      "upstream_bind_config": {
       "source_address": {
        "address": "192.168.5.14",
        "port_value": 0
       },
       "socket_options": [
        {
         "name": "1",
         "int_value": "252"
        }
       ]
      },
      "transport_socket": {
       "name": "tls",
       "typed_config": {
        "@type": "type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext",
        "common_tls_context": {
         "alpn_protocols": [
          "h2"
         ]
        }
       }
      }
     },
     "last_updated": "2020-06-12T13:51:04.562Z"
    },
    {
     "version_info": "301",
     "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "UI-Nudm_NF-udm_instance_1_FQDN-udm01.udm.5gc.mnc03.mcc262.3gppnetwork.org_aggregate",
      "connect_timeout": "5s",
      "lb_policy": "CLUSTER_PROVIDED",
      "http2_protocol_options": {
       "hpack_table_size": 4096
      },
      "cluster_type": {
       "name": "envoy.clusters.aggregate",
       "typed_config": {
        "@type": "type.googleapis.com/envoy.config.cluster.aggregate.v2alpha.ClusterConfig",
        "clusters": [
         "UI-Nudm_NF-udm_instance_1_FQDN-udm01.udm.5gc.mnc03.mcc262.3gppnetwork.org_192.168.5.14",
         "UI-Nudm_NF-udm_instance_1_FQDN-udm01.udm.5gc.mnc03.mcc262.3gppnetwork.org_fc00::192:168:5:e"
        ]
       }
      }
     },
     "last_updated": "2020-06-16T11:48:36.453Z"
    },
    {
     "version_info": "3",
     "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "UI-Nudm_NF-udm_instance_1_FQDN-udm01.udm.5gc.mnc03.mcc262.3gppnetwork.org_fc00::192:168:5:e",
      "type": "EDS",
      "eds_cluster_config": {
       "eds_config": {
        "ads": {}
       }
      },
      "connect_timeout": "5s",
      "lb_policy": "LEAST_REQUEST",
      "circuit_breakers": {
       "thresholds": [
        {
         "max_requests": 10000,
         "max_retries": 3
        },
        {
         "priority": "HIGH",
         "max_requests": 10000,
         "max_retries": 3
        }
       ]
      },
      "http2_protocol_options": {
       "hpack_table_size": 4096
      },
      "upstream_bind_config": {
       "source_address": {
        "address": "fc00::192:168:5:e",
        "port_value": 0
       },
       "socket_options": [
        {
         "name": "1",
         "int_value": "252"
        }
       ]
      },
      "transport_socket": {
       "name": "tls",
       "typed_config": {
        "@type": "type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext",
        "common_tls_context": {
         "alpn_protocols": [
          "h2"
         ]
        }
       }
      }
     },
     "last_updated": "2020-06-12T13:51:04.593Z"
    },

Thanks !

[snowp]

I don't see any health checking config - how are you expecting Envoy to notice that the first endpoint is unreachable? There might be something else going on here.

Mind including the output of the /clusters admin endpoint and the cluster stats for the clusters involved here?

[chrikar]

Hello @snowp,

Thanks for the help!

We're working together on this with @vinzo99.

I don't see any health checking config - how are you expecting Envoy to notice that the first endpoint is unreachable? There might be something else going on here.

We've been relying on the passive health checking using the circuit breaking functionality for that. Do we also need active health checking for this to work?

We've also been experimenting with the front-envoy example to see how the aggregate cluster and its failover funcionality works, and the following example seems to be working for the aggregated clusters where the docker of service1 is stopped and the traffic is routed to service2.

Mind including the output of the /clusters admin endpoint and the cluster stats for the clusters involved here?

We've been using the following testcase, as we're trying to use the aggregate cluster feature to be able to failover between two different networks. One IPv4 and one IPv6. In both cases the target host is only reachable through IPv4. When the first cluster defined in the aggregate cluster is IPv4, then we get traffic to pass. When the first cluster is IPv6 (which is expected to fail and then use the secondary cluster) we can't get any traffic through.

Passing case - IPv4 cluster is primary

test_cluster_aggregate::default_priority::max_connections::1024
test_cluster_aggregate::default_priority::max_pending_requests::1024
test_cluster_aggregate::default_priority::max_requests::1024
test_cluster_aggregate::default_priority::max_retries::3
test_cluster_aggregate::high_priority::max_connections::1024
test_cluster_aggregate::high_priority::max_pending_requests::1024
test_cluster_aggregate::high_priority::max_requests::1024
test_cluster_aggregate::high_priority::max_retries::3
test_cluster_aggregate::added_via_api::true

test_cluster_192.168.5.14::default_priority::max_connections::1024
test_cluster_192.168.5.14::default_priority::max_pending_requests::1024
test_cluster_192.168.5.14::default_priority::max_requests::10000
test_cluster_192.168.5.14::default_priority::max_retries::3
test_cluster_192.168.5.14::high_priority::max_connections::1024
test_cluster_192.168.5.14::high_priority::max_pending_requests::1024
test_cluster_192.168.5.14::high_priority::max_requests::10000
test_cluster_192.168.5.14::high_priority::max_retries::3
test_cluster_192.168.5.14::added_via_api::true
test_cluster_192.168.5.14::192.168.5.4:443::cx_active::0
test_cluster_192.168.5.14::192.168.5.4:443::cx_connect_fail::0
test_cluster_192.168.5.14::192.168.5.4:443::cx_total::1
test_cluster_192.168.5.14::192.168.5.4:443::rq_active::0
test_cluster_192.168.5.14::192.168.5.4:443::rq_error::1
test_cluster_192.168.5.14::192.168.5.4:443::rq_success::2
test_cluster_192.168.5.14::192.168.5.4:443::rq_timeout::0
test_cluster_192.168.5.14::192.168.5.4:443::rq_total::3
test_cluster_192.168.5.14::192.168.5.4:443::hostname::
test_cluster_192.168.5.14::192.168.5.4:443::health_flags::healthy
test_cluster_192.168.5.14::192.168.5.4:443::weight::1
test_cluster_192.168.5.14::192.168.5.4:443::region::
test_cluster_192.168.5.14::192.168.5.4:443::zone::
test_cluster_192.168.5.14::192.168.5.4:443::sub_zone::
test_cluster_192.168.5.14::192.168.5.4:443::canary::false
test_cluster_192.168.5.14::192.168.5.4:443::priority::0
test_cluster_192.168.5.14::192.168.5.4:443::success_rate::-1.0
test_cluster_192.168.5.14::192.168.5.4:443::local_origin_success_rate::-1.0

test_cluster_fc00::192:168:5:e::default_priority::max_connections::1024
test_cluster_fc00::192:168:5:e::default_priority::max_pending_requests::1024
test_cluster_fc00::192:168:5:e::default_priority::max_requests::10000
test_cluster_fc00::192:168:5:e::default_priority::max_retries::3
test_cluster_fc00::192:168:5:e::high_priority::max_connections::1024
test_cluster_fc00::192:168:5:e::high_priority::max_pending_requests::1024
test_cluster_fc00::192:168:5:e::high_priority::max_requests::10000
test_cluster_fc00::192:168:5:e::high_priority::max_retries::3
test_cluster_fc00::192:168:5:e::added_via_api::true

-------------------------------------------------------------------------------------------------------------------------------

Failing case - IPv6 cluster is primary

test_cluster_aggregate::default_priority::max_connections::1024
test_cluster_aggregate::default_priority::max_pending_requests::1024
test_cluster_aggregate::default_priority::max_requests::1024
test_cluster_aggregate::default_priority::max_retries::3
test_cluster_aggregate::high_priority::max_connections::1024
test_cluster_aggregate::high_priority::max_pending_requests::1024
test_cluster_aggregate::high_priority::max_requests::1024
test_cluster_aggregate::high_priority::max_retries::3
test_cluster_aggregate::added_via_api::true

test_cluster_192.168.5.14::default_priority::max_connections::1024
test_cluster_192.168.5.14::default_priority::max_pending_requests::1024
test_cluster_192.168.5.14::default_priority::max_requests::10000
test_cluster_192.168.5.14::default_priority::max_retries::3
test_cluster_192.168.5.14::high_priority::max_connections::1024
test_cluster_192.168.5.14::high_priority::max_pending_requests::1024
test_cluster_192.168.5.14::high_priority::max_requests::10000
test_cluster_192.168.5.14::high_priority::max_retries::3
test_cluster_192.168.5.14::added_via_api::true
test_cluster_192.168.5.14::192.168.5.4:443::cx_active::0
test_cluster_192.168.5.14::192.168.5.4:443::cx_connect_fail::0
test_cluster_192.168.5.14::192.168.5.4:443::cx_total::1
test_cluster_192.168.5.14::192.168.5.4:443::rq_active::0
test_cluster_192.168.5.14::192.168.5.4:443::rq_error::1
test_cluster_192.168.5.14::192.168.5.4:443::rq_success::2
test_cluster_192.168.5.14::192.168.5.4:443::rq_timeout::0
test_cluster_192.168.5.14::192.168.5.4:443::rq_total::3
test_cluster_192.168.5.14::192.168.5.4:443::hostname::
test_cluster_192.168.5.14::192.168.5.4:443::health_flags::healthy
test_cluster_192.168.5.14::192.168.5.4:443::weight::1
test_cluster_192.168.5.14::192.168.5.4:443::region::
test_cluster_192.168.5.14::192.168.5.4:443::zone::
test_cluster_192.168.5.14::192.168.5.4:443::sub_zone::
test_cluster_192.168.5.14::192.168.5.4:443::canary::false
test_cluster_192.168.5.14::192.168.5.4:443::priority::0
test_cluster_192.168.5.14::192.168.5.4:443::success_rate::-1.0
test_cluster_192.168.5.14::192.168.5.4:443::local_origin_success_rate::-1.0

test_cluster_fc00::192:168:5:e::default_priority::max_connections::1024
test_cluster_fc00::192:168:5:e::default_priority::max_pending_requests::1024
test_cluster_fc00::192:168:5:e::default_priority::max_requests::10000
test_cluster_fc00::192:168:5:e::default_priority::max_retries::3
test_cluster_fc00::192:168:5:e::high_priority::max_connections::1024
test_cluster_fc00::192:168:5:e::high_priority::max_pending_requests::1024
test_cluster_fc00::192:168:5:e::high_priority::max_requests::10000
test_cluster_fc00::192:168:5:e::high_priority::max_retries::3
test_cluster_fc00::192:168:5:e::added_via_api::true

[vinzo99]

Hello,
Is info provided by @Christoska sufficient for you to troubleshoot this issue ?
Is there anything more we can do to help on this ?
Thanks

[snowp]

In both those cases there only seems to be IPv4 addresses present? The Ipv6 cluster doesn't appear to have any endpoints.

Re passive health checking and circuit breaking: hitting circuit breakers does not mark endpoints as unhealthy, and so they wouldn't trigger failover. If you're using the outlier detection mechanism to do passive health checking that should work, though might not be perfect (endpoints could be returned to healthy at times causing more failures).

An option to test out (assuming you can ensure that the Ipv6 cluster has endpoints) is to use TCP active health checking, in which case it will treat endpoints as unhealthy unless it can maintain a TCP connection to them. This would be a bit more robust, though I'm not sure if that's your problem here.

[chrikar]

Thank you for the reply @snowp!

We'll give active health check a try and see if it works.

Could this be related to #11498 ?
As we're using the go-control-plane's xds_server to dynamically change the cluster order in the aggregate cluster.