Request fails during ai gateway upgrade

[yuzisun]

Description:
During upgrade we saw requests failed with the following errors, seems like envoy is able to accept requests but ai gateway extproc container uds socket is not ready yet.

ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_re
mote_connection_failure,_transport_failure_reason:_immediate_connect_error:_No_such_file_or_directory|remot
e_address:/etc/ai-gateway-extproc-uds/run.sock

Repro steps:

• Continuously sending requests to envoy ai gateway
• Upgrade envoy ai gateway helm chart
• Observed 500 errors during upgrade

[mathetake]

this feels very weird to me. The envoy pod will have readiness probe invoking the health check endpoint of extproc container, which in turns checks the connection to the gRPC server's listener. So by the time when the pod marked ready, I thought it's 100% ready but it seems not working that way?

One thing maybe happening is that

• Envoy tries to check the connectivity before UDS is ready, and considers the extproc UDS not ready for a couple of seconds (like health checking interval by Envoy).
• ExtProc on the other hand will be considered ready at k8s level as the readiness probe is only considering about the in-process connection
• As as result, there might be a gap between the time when Envoy tries again to do the health check on the extproc cluster and the time when extproc becomes ready.

So, I think one thing you could do is to add another health check in Envoy container (not extproc) to invoke the /some/random/non-existing/path and then it should get 404 from the extproc with "unsupported path".

ai-gateway/internal/extproc/server.go

Lines 193 to 194 in 8d3310e

	Body: fmt.Appendf(nil, "unsupported path: %s", path),
	GrpcStatus: &extprocv3.GrpcStatus{Status: uint32(codes.NotFound)},

[mathetake]

This is relevant to #1060

[mathetake]

I got a repro #1248 -- the observation is that it's not a new pod that returns the error but the old one during the termination. Looks like extproc exits before envoy stops receiving requests