Envoy proxy 500 direct_response when HTTPRoute and Gateway is accepted and resolved with success.

[flux-ricky]

We are running Envoy Gateway (v1.4) and making use of HTTPRoute for proxying a single domain to two different public domains. Our Gateway and GatewayClass are configured and accepted, HTTPRoute shows resolved and accepted, nothing in the resources we have configured show any issues. And yet Envoy will return a 500 error as a "direct_response".

Please see the configuration below:

HTTP Route manifests

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: the-service-migration-proxy
  namespace: client1-app
spec:
  hostnames:
  - the-service-client1-app-gateway.customer.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: public
    namespace: envoy-gateway-system
    sectionName: https
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: the-service-knative-migration-service
      port: 443
      weight: 100
    filters:
    - type: URLRewrite
      urlRewrite:
        hostname: client1-demo-the.customer.example.com
    matches:
    - path:
        type: PathPrefix
        value: /
    timeouts:
      backendRequest: 120s
      request: 120s
status:
  parents:
  - conditions:
    - lastTransitionTime: "2025-06-03T04:12:43Z"
      message: Route is accepted
      observedGeneration: 3
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-06-03T04:12:43Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 3
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: public
      namespace: envoy-gateway-system
      sectionName: https
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  name: the-service-knative-migration-service
  namespace: client1-app
spec:
  clusterIP: 172.20.130.132
  clusterIPs:
  - 172.20.130.132
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
---
addressType: FQDN
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
  - client1-the-service.customer.demofluxto.me
  conditions: {}
kind: EndpointSlice
metadata:
  labels:
    kubernetes.io/service-name: knative-migration-service
  name: the-service-knative-migration-service
  namespace: client1-app
ports:
- name: https
  port: 443
  protocol: TCP

Envoy Gateway Resources

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: envoy-gateway-system
spec:
  gatewayClassName: public
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    hostname: '*.customer.example.com'
    name: https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: cluster-wildcard-cert
      mode: Terminate
status:
  addresses:
  - type: Hostname
    value: k8s-envoygat-envoyenv-48d4620ca2-cecb5a5dc9017074.elb.ap-southeast-2.amazonaws.com
  conditions:
  - lastTransitionTime: "2025-06-02T19:20:34Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2025-06-02T19:20:34Z"
    message: Address assigned to the Gateway, 1/1 envoy replicas available
    observedGeneration: 1
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2025-06-02T19:20:25Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2025-06-02T19:20:25Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-06-02T19:20:25Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: https
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  finalizers:
  - gateway-exists-finalizer.gateway.networking.k8s.io
  name: public
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: public-gateway-config
    namespace: envoy-gateway-system
status:
  conditions:
  - lastTransitionTime: "2025-01-29T00:28:35Z"
    message: Valid GatewayClass
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: public-gateway-config
  namespace: envoy-gateway-system
spec:
  logging:
    level:
      default: warn
  provider:
    kubernetes:
      envoyDeployment:
        container:
          image: 340501053493.dkr.ecr.ap-southeast-2.amazonaws.com/registry-1.docker.io/envoyproxy/envoy:distroless-v1.34.1
          resources:
            limits:
              cpu: 100m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 512Mi
        pod:
          affinity:
            nodeAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - preference:
                  matchExpressions:
                  - key: example.com/nodegroup
                    operator: In
                    values:
                    - general
                weight: 100
          tolerations:
          - effect: NoExecute
            key: mng-node
            operator: Exists
          topologySpreadConstraints:
          - labelSelector:
              matchLabels:
                gateway.envoyproxy.io/owning-gateway-name: public
            maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
          - labelSelector:
              matchLabels:
                gateway.envoyproxy.io/owning-gateway-name: public
            maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: ScheduleAnyway
        replicas: 1
      envoyService:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "5"
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /ready
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "19001"
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "2"
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: load_balancing.cross_zone.enabled=true
          service.beta.kubernetes.io/aws-load-balancer-type: external
        externalTrafficPolicy: Local
        type: LoadBalancer
    type: Kubernetes
  telemetry:
    tracing:
      provider:
        host: daemonset-collector.opentelemetry-collectors.svc.cluster.local
        port: 4317
        type: OpenTelemetry
      samplingRate: 100

Dumping the config from the running envoy proxy shows the direct response configured.

EnvoyProxy Config Dump

{
        "name": "envoy-gateway-system/public/https/the-service-client1-app-gateway_customer_example_com",
        "domains": [
         "the-service-client1-app-gateway.customer.example.com"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/"
          },
          "metadata": {
           "filter_metadata": {
            "envoy-gateway": {
             "resources": [
              {
               "namespace": "client1-app",
               "name": "the-service-migration-proxy",
               "kind": "HTTPRoute"
              }
             ]
            }
           }
          },
          "direct_response": {
           "status": 500
          },
          "name": "httproute/client1-app/the-service-migration-proxy/rule/0/match/0/the-service-client1-app-gateway_customer_example_com"
         }
        ],
        "metadata": {
         "filter_metadata": {
          "envoy-gateway": {
           "resources": [
            {
             "kind": "Gateway",
             "name": "public",
             "namespace": "envoy-gateway-system",
             "sectionName": "https"
            }
           ]
          }
         }
}

I am perplexed as to why this is happening, especially as there is no issue that I can see to help debug. Is there somewhere else I can look to try and understand what is going on?

Any help appreciated.

[zirain]

can you check the HTTPRoute with kubectl get HTTPRoute -n client1-metric metrics-service-migration-proxy

[flux-ricky]

I should mention a few things:

• This behaviour has been seen on all the versions I've tried: 1.2.5, 1.3.0, 1.3.2
• Removing the BakendTlsPolicy has the same behaviour.
• However, at some point this was working for us, and at some point in refactoring we've ended up in this scenario.

[arkodg]

you may need to filter out config to see which one is causing this
Envoy Gateway only adds the 500 direct response if there's a configuration issue and marks the issue in the resource status
https://gateway.envoyproxy.io/docs/troubleshooting/configuration/

[flux-ricky]

We found that after our refactoring our EndpointSlice label kubernetes.io/service-name: knative-migration-service didn't match the Service name: the-service-knative-migration-service. 🤦

While I understand that this was totally our problem with misconfiguring the EndpointSlice + Service, it appears that EnvoyGateway was smart enough to figure this out and provision the static 500 response. I am wondering what you think about surfacing this to the end user, just like various others issues are currently?

Unfortunately nothing was surfaced on the HTTPRoute, Service, or EndpointSlice to give us a hint there was a problem.

Many thanks for your time.

[arkodg]

glad you figured this out, can you raise a GH issue to track this, we should have had a a status in the route for it

[flux-ricky]

No problem, here it is @arkodg #6269

I will close this discussion now.

[flux-ricky]

Issue created for this here: #6269