Decouple endpoint updates from listener and route updates

[arkodg]

Description:

Describe the issue.

We've seen in #7394 that translation config involving slow or faulty I/O associated with remote configuration (jwt, oidc, wasm) slows down the entire translation

This impacts common update flows such as deployment rollout and slows it down.

We need to decouple endpoint updates from the rest of the updates which is more critical when dealing with live traffic

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[arkodg]

this is a little tricky to solve today, we fetch remote configuration in the gateway-api layer because we want to update the status with pass/fail information. If we could move fetching remote configuration to the xds layer, we could easily parallize work to process endpoints and other config separately
cc @envoyproxy/gateway-maintainers

[zhaohuabing]

Moving the fetching logic from gateway-api to xds layer will help, but it could also slow down HTTPRoute propagation unless we also process routes with remote configurations in separate goroutines.

[zhaohuabing]

Another option that I'm thinking about is to fetch these remote configurations in a background goroutine, and re-trigger the Gateway API translator once they're ready.

[arkodg]

Another option that I'm thinking about is to fetch these remote configurations in a background goroutine, and re-trigger the Gateway API translator once they're ready.

@zhaohuabing this will cause policy flaps

[zhaohuabing]

We can try the first time in the Gateway API translator, and retry in the background goroutine if failed. That would avoid flaps but introduce delay at the first translation of a remote configuration.