Ingressgateway cache pod ip although pod is deleted, access with this IP and bring 503 for several hours.

[johnzheng1975]

Title: One line description
Ingressgateway cache pod ip although pod is deleted, access with this IP and bring 503 for several hours.
After delete, the IP is removed. (from cilium log, from istiod log)
The IP is not in Istio EDS. The IP is in http://localhost:15000/clusters

Description:

What issue is being seen? Describe what should be happening instead of
the bug, for example: Envoy should not crash, the expected value isn't
returned, etc.
old IP still exits (10.203.103.174 10.203.128.80 are the old IP belong to terminated pods)

 2002  k exec -ti -n is istio-ingressgateway-apigee-7d77dd685-dzjw4  --  curl -X POST http://localhost:15000/clusters | grep roles > clusters_istio-ingressgateway-apigee-7d77dd685-dzjw4.txt
 
 2004  k exec -ti -n is  istio-ingressgateway-apigee-7d77dd685-mzfn6  --  curl -X POST http://localhost:15000/clusters | grep roles > clusters_istio-ingressgateway-apigee-7d77dd685-mzfn6.txt

 2005  k exec -ti -n is  istio-ingressgateway-apigee-7d77dd685-plp59  --  curl -X POST http://localhost:15000/clusters | grep roles > clusters_istio-ingressgateway-apigee-7d77dd685-plp59.txt


outbound|80||roles.relationship.svc.cluster.local::observability_name::outbound|80||roles.relationship.svc.cluster.local;
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_connections::4294967295
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_pending_requests::4294967295
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_requests::4294967295
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_retries::4294967295
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_connections::1024
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_requests::1024
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_retries::3
outbound|80||roles.relationship.svc.cluster.local::added_via_api::true
outbound|80||roles.relationship.svc.cluster.local::eds_service_name::outbound|80||roles.relationship.svc.cluster.local
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::cx_active::2
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::cx_connect_fail::66
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::cx_total::68
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_active::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_error::92
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_success::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_timeout::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_total::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::hostname::
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::health_flags::healthy
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::weight::1
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::region::us-west-2
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::zone::us-west-2a
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::sub_zone::
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::canary::false
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::priority::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::success_rate::-1
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::local_origin_success_rate::-1
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::cx_active::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::cx_connect_fail::64
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::cx_total::64
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_active::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_error::87
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_success::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_timeout::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_total::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::hostname::
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::health_flags::healthy
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::weight::1
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::region::us-west-2
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::zone::us-west-2b
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::sub_zone::
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::canary::false
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::priority::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::success_rate::-1
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::local_origin_success_rate::-1

Repro steps:

Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.
When hundreds pods deleted/ created on the same time, this will happen easily.

[johnzheng1975]

After hundreds pod start, seems some stale IP will remain in ingress gateway.

Firstly, I run command as istio/istio#54081 (comment), cannot find root reason.

 1804  ./istioctl  pc eds istio-ingressgateway-apigee-7d77dd685-dzjw4 -n istio-system > eds-istio-ingressgateway-apigee-7d77dd685-dzjw4.txt
 1806  ./istioctl  pc eds istio-ingressgateway-apigee-7d77dd685-mzfn6  -n istio-system > eds-istio-ingressgateway-apigee-7d77dd685-mzfn6.txt
 1807  ./istioctl  pc eds istio-ingressgateway-apigee-7d77dd685-plp59   -n istio-system > eds-istio-ingressgateway-apigee-7d77dd685-plp59.txt


./istioctl  x internal-debug endpointShardz > endpointShardz.txt


./istioctl proxy-status > proxy-status.txt

From the result, all new pod ip is showed (10.203.130.243/ 10.203.191.77), all old pod ip (10.203.103.174/ 10.203.128.80) cannot be found in Istio eds log, in endpointShardz.txt, in proxy-status.txt. (It may means istio works as expected)

I delete all pods under kube-system, to refresh coredns, aws cni, cilium, kube-proxy, not work. 503 still raise. (It may means not aws cni issue, not coredns issue, not cilium issue, not kube-proxy issue.)

In the node of istio-ingressgateway, I check iptables, no problem. (It may means: iptable no issues)

sudo iptables-save | grep '10.203.130.243'
-- has values

sudo iptables-save | grep '10.203.103.174'
-- no result.

coredns-1.log
coredns-2.log
eds-istio-ingressgateway-apigee-7d77dd685-dzjw4.txt
eds-istio-ingressgateway-apigee-7d77dd685-mzfn6.txt
eds-istio-ingressgateway-apigee-7d77dd685-plp59.txt
endpointShardz.txt
istio-ingressgateway-apigee-7d77dd685-dzjw4.istio-system.log
proxy-status.txt
proxy-status-1.txt

[johnzheng1975]

Finally, I run below, found the root reason: old IP still exits (10.203.103.174 10.203.128.80 are the old IP belong to terminated pods)

You can find: health_flags::healthy although :rq_error::92

Expect Result: these clusters should be removed.
Actual Result: these will keep there for 12+ hours, until istiod/ingress/service pod is created/deleted. on that time the these will be refreshed.

 2002  k exec -ti -n is istio-ingressgateway-apigee-7d77dd685-dzjw4  --  curl -X POST http://localhost:15000/clusters | grep roles > clusters_istio-ingressgateway-apigee-7d77dd685-dzjw4.txt
 
 2004  k exec -ti -n is  istio-ingressgateway-apigee-7d77dd685-mzfn6  --  curl -X POST http://localhost:15000/clusters | grep roles > clusters_istio-ingressgateway-apigee-7d77dd685-mzfn6.txt

 2005  k exec -ti -n is  istio-ingressgateway-apigee-7d77dd685-plp59  --  curl -X POST http://localhost:15000/clusters | grep roles > clusters_istio-ingressgateway-apigee-7d77dd685-plp59.txt


outbound|80||roles.relationship.svc.cluster.local::observability_name::outbound|80||roles.relationship.svc.cluster.local;
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_connections::4294967295
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_pending_requests::4294967295
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_requests::4294967295
outbound|80||roles.relationship.svc.cluster.local::default_priority::max_retries::4294967295
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_connections::1024
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_requests::1024
outbound|80||roles.relationship.svc.cluster.local::high_priority::max_retries::3
outbound|80||roles.relationship.svc.cluster.local::added_via_api::true
outbound|80||roles.relationship.svc.cluster.local::eds_service_name::outbound|80||roles.relationship.svc.cluster.local
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::cx_active::2
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::cx_connect_fail::66
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::cx_total::68
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_active::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_error::92
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_success::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_timeout::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::rq_total::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::hostname::
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::health_flags::healthy
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::weight::1
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::region::us-west-2
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::zone::us-west-2a
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::sub_zone::
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::canary::false
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::priority::0
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::success_rate::-1
outbound|80||roles.relationship.svc.cluster.local::10.203.103.174:8080::local_origin_success_rate::-1
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::cx_active::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::cx_connect_fail::64
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::cx_total::64
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_active::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_error::87
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_success::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_timeout::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::rq_total::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::hostname::
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::health_flags::healthy
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::weight::1
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::region::us-west-2
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::zone::us-west-2b
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::sub_zone::
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::canary::false
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::priority::0
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::success_rate::-1
outbound|80||roles.relationship.svc.cluster.local::10.203.128.80:8080::local_origin_success_rate::-1

clusters_istio-ingressgateway-apigee-7d77dd685-dzjw4.txt
clusters_istio-ingressgateway-apigee-7d77dd685-mzfn6.txt
clusters_istio-ingressgateway-apigee-7d77dd685-plp59.txt

[johnzheng1975]

You can also refer: istio/istio#54081 (comment)
Thanks a lot.

[phlax]

cc @botengyao @yanavlasov @zuercher

[yanavlasov]

This problem has not been diagnosed. Please complete diagnostics in the istio/istio#54081 to determine if this is a problem in Envoy.

[johnzheng1975]

We still do not know: Istio notify envoy or not for terminate pod.
Maybe istio can fix it in its side since restart istiod can recover this -- @ramaraochavali is working on this.

However, not sure envoy can recover this directly, based on error requests and mark it unhealthy. Thanks a lot.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.