Unable to get gateway to listen on port 80 (containerPorts are not updated) #5707

finomen · 2025-04-10T10:03:58Z

finomen
Apr 10, 2025

I'm trying to install envoy proxy on baremetal cluster with cilium CNI.

Looks like it does not add listeners to pods for gateway ports.

EnvoyProxy
`
Name: proxy-config
Namespace: envoy-gateway-system
Labels: kustomize.toolkit.fluxcd.io/name=infrastructure
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:
API Version: gateway.envoyproxy.io/v1alpha1
Kind: EnvoyProxy
Metadata:
Creation Timestamp: 2025-04-10T09:07:26Z
Generation: 3
Resource Version: 38778896
UID: aecdad3b-a375-4d49-a3e7-c3058d432958
Spec:
Logging:
Level:
Default: warn
Merge Gateways: true
Provider:
Kubernetes:
Envoy Daemon Set:
Envoy Service:
Annotations:
lbipam.cilium.io/ips: XX.XX.XX.XX,YY.YY.YY.YY
lbipam.cilium.io/sharing-cross-namespace: *
lbipam.cilium.io/sharing-key: share
External Traffic Policy: Cluster
Type: LoadBalancer
Type: Kubernetes
Routing Type: Service
Events:

`

GatewayClass:
kubectl describe gatewayclass Name: envoy Namespace: Labels: kustomize.toolkit.fluxcd.io/name=infrastructure kustomize.toolkit.fluxcd.io/namespace=flux-system Annotations: <none> API Version: gateway.networking.k8s.io/v1 Kind: GatewayClass Metadata: Creation Timestamp: 2025-04-10T09:41:54Z Finalizers: gateway-exists-finalizer.gateway.networking.k8s.io Generation: 1 Resource Version: 38782620 UID: e47daff1-17cb-458e-92b3-5d877659741e Spec: Controller Name: gateway.envoyproxy.io/gatewayclass-controller Parameters Ref: Group: gateway.envoyproxy.io Kind: EnvoyProxy Name: proxy-config Namespace: envoy-gateway-system Status: Conditions: Last Transition Time: 2025-04-10T09:41:54Z Message: Valid GatewayClass Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Events: <none>
Getway
Name: default-gw Namespace: envoy-gateway-system Labels: kustomize.toolkit.fluxcd.io/name=infrastructure kustomize.toolkit.fluxcd.io/namespace=flux-system Annotations: <none> API Version: gateway.networking.k8s.io/v1 Kind: Gateway Metadata: Creation Timestamp: 2025-04-10T09:49:34Z Generation: 1 Resource Version: 38783081 UID: a0374adf-14bc-4b2d-a124-8246521b9e28 Spec: Gateway Class Name: envoy Listeners: Allowed Routes: Namespaces: From: Same Hostname: example.com Name: http Port: 80 Protocol: HTTP Status: Addresses: Type: IPAddress Value: XX.XX.XX.XX Type: IPAddress Value: YY.YY.YY.YY Conditions: Last Transition Time: 2025-04-10T09:50:16Z Message: The Gateway has been scheduled by Envoy Gateway Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2025-04-10T09:50:16Z Message: Address assigned to the Gateway, 4/4 envoy replicas available Observed Generation: 1 Reason: Programmed Status: True Type: Programmed Listeners: Attached Routes: 1 Conditions: Last Transition Time: 2025-04-10T09:49:34Z Message: Sending translated listener configuration to the data plane Observed Generation: 1 Reason: Programmed Status: True Type: Programmed Last Transition Time: 2025-04-10T09:49:34Z Message: Listener has been successfully translated Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2025-04-10T09:49:34Z Message: Listener references have been resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Name: http Supported Kinds: Group: gateway.networking.k8s.io Kind: HTTPRoute Group: gateway.networking.k8s.io Kind: GRPCRoute Events: <none>
Service
Name: envoy-envoy-84532b30 Namespace: envoy-gateway-system Labels: app.kubernetes.io/component=proxy app.kubernetes.io/managed-by=envoy-gateway app.kubernetes.io/name=envoy gateway.envoyproxy.io/owning-gatewayclass=envoy Annotations: lbipam.cilium.io/ips: XX.XX.XX.XX,YY.YY.YY.YY lbipam.cilium.io/sharing-cross-namespace: * lbipam.cilium.io/sharing-key: share Selector: app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gatewayclass=envoy Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.97.76.75 IPs: 10.97.76.75 LoadBalancer Ingress: XX.XX.XX.XX,YY.YY.YY.YY Port: http-80 80/TCP TargetPort: 10080/TCP NodePort: http-80 31576/TCP Endpoints: 10.244.0.147:10080,10.244.1.3:10080,10.244.2.160:10080 + 1 more... Session Affinity: None External Traffic Policy: Cluster Events: <none>

Pod (one of daemonset, they are identical)
`
Name: envoy-envoy-84532b30-dflh4
Namespace: envoy-gateway-system
Priority: 0
Service Account: envoy-envoy-84532b30
Node: fsn-ctl-2/172.16.0.3
Start Time: Thu, 10 Apr 2025 11:49:35 +0200
Labels: app.kubernetes.io/component=proxy
app.kubernetes.io/managed-by=envoy-gateway
app.kubernetes.io/name=envoy
controller-revision-hash=66c6968d5b
gateway.envoyproxy.io/owning-gatewayclass=envoy
pod-template-generation=1
Annotations: prometheus.io/path: /stats/prometheus
prometheus.io/port: 19001
prometheus.io/scrape: true
Status: Running
IP: 10.244.2.160
IPs:
IP: 10.244.2.160
IP: 2001:db8:42::717f
Controlled By: DaemonSet/envoy-envoy-84532b30
Containers:
envoy:
Container ID: containerd://ab0bced6fa8523be72b5418eae0913a3e73320dc064cd71f391c196b6faa4e73
Image: docker.io/envoyproxy/envoy:distroless-v1.33.1
Image ID: docker.io/envoyproxy/envoy@sha256:42653673bbc413c41c545ce6f134e0847d88c44932fc3c8e5d3b0907b36ffa31
Port: 19001/TCP
Host Port: 0/TCP
SeccompProfile: RuntimeDefault
Command:
envoy
Args:
--service-cluster envoy
--service-node $(ENVOY_POD_NAME)
--config-yaml admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 19000
layered_runtime:
layers:
- name: global_config
static_layer:
envoy.restart_features.use_eds_cache_for_ads: true
re2.max_program_size.error_level: 4294967295
re2.max_program_size.warn_level: 1000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
listeners:
- name: envoy-gateway-proxy-ready-0.0.0.0-19001
address:
socket_address:
address: '0.0.0.0'
port_value: 19001
protocol: TCP
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: eg-ready-http
normalize_path: true
route_config:
name: local_route
virtual_hosts:
- name: prometheus_stats
domains:
- "*"
routes:
- match:
path: /stats/prometheus
headers:
- name: ":method"
exact_match: GET
route:
cluster: prometheus_stats
http_filters:
- name: envoy.filters.http.health_check
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
pass_through_mode: false
headers:
- name: ":path"
string_match:
exact: /ready
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: prometheus_stats
connect_timeout: 0.250s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: prometheus_stats
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 19000
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- load_balancing_weight: 1
lb_endpoints:
- load_balancing_weight: 1
endpoint:
address:
socket_address:
address: envoy-gateway.envoy-gateway-system.svc.cluster.local
port_value: 18000
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
explicit_http_config:
http2_protocol_options:
connection_keepalive:
interval: 30s
timeout: 5s
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: /sds/xds-certificate.json
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: /sds/xds-trusted-ca.json
resource_api_version: V3
- name: wasm_cluster
type: STRICT_DNS
connect_timeout: 10s
load_assignment:
cluster_name: wasm_cluster
endpoints:
- load_balancing_weight: 1
lb_endpoints:
- load_balancing_weight: 1
endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18002
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
explicit_http_config:
http2_protocol_options: {}
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: /sds/xds-certificate.json
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: /sds/xds-trusted-ca.json
resource_api_version: V3
overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: "envoy.resource_monitors.global_downstream_max_connections"
typed_config:
"@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
max_active_downstream_connections: 50000

  --log-level warn
  --cpuset-threads
  --drain-strategy immediate
  --drain-time-s 60
State:          Running
  Started:      Thu, 10 Apr 2025 11:49:36 +0200
Ready:          True
Restart Count:  0
Requests:
  cpu:      100m
  memory:   512Mi
Readiness:  http-get http://:19001/ready delay=0s timeout=1s period=5s #success=1 #failure=1
Startup:    http-get http://:19001/ready delay=0s timeout=1s period=10s #success=1 #failure=30
Environment:
  ENVOY_GATEWAY_NAMESPACE:  envoy-gateway-system (v1:metadata.namespace)
  ENVOY_POD_NAME:           envoy-envoy-84532b30-dflh4 (v1:metadata.name)
Mounts:
  /certs from certs (ro)
  /sds from sds (rw)

shutdown-manager:
Container ID: containerd://ab6414c1a7f9bb19cbf08366870625e448ae1d87af595a48f0ed1ae6c4b1fe04
Image: docker.io/envoyproxy/gateway:v1.3.2
Image ID: docker.io/envoyproxy/gateway@sha256:d6e5e3c7291e246f3c13311b640dc8a475dfaefe7961759e1dc2b622a8f9c1a5
Port:
Host Port:
SeccompProfile: RuntimeDefault
Command:
envoy-gateway
Args:
envoy
shutdown-manager
State: Running
Started: Thu, 10 Apr 2025 11:49:36 +0200
Ready: True
Restart Count: 0
Requests:
cpu: 10m
memory: 32Mi
Liveness: http-get http://:19002/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:19002/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Startup: http-get http://:19002/healthz delay=0s timeout=1s period=10s #success=1 #failure=30
Environment:
ENVOY_GATEWAY_NAMESPACE: envoy-gateway-system (v1:metadata.namespace)
ENVOY_POD_NAME: envoy-envoy-84532b30-dflh4 (v1:metadata.name)
Mounts:
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
certs:
Type: Secret (a volume populated by a Secret)
SecretName: envoy
Optional: false
sds:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: envoy-envoy-84532b30
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message

Normal Scheduled 11m default-scheduler Successfully assigned envoy-gateway-system/envoy-envoy-84532b30-dflh4 to fsn-ctl-2
Normal Pulled 11m kubelet Container image "docker.io/envoyproxy/envoy:distroless-v1.33.1" already present on machine
Normal Created 11m kubelet Created container: envoy
Normal Started 11m kubelet Started container envoy
Normal Pulled 11m kubelet Container image "docker.io/envoyproxy/gateway:v1.3.2" already present on machine
Normal Created 11m kubelet Created container: shutdown-manager
Normal Started 11m kubelet Started container shutdown-manager
Warning Unhealthy 10m (x3 over 11m) kubelet Startup probe failed: Get "http://10.244.2.160:19001/ready": dial tcp 10.244.2.160:19001: connect: connection refused
`

arkodg · 2025-04-11T12:36:18Z

arkodg
Apr 11, 2025
Maintainer

the containerPort is an internal detail, if you'd like it to be the same as listener port, you can set it with useListenerPortAsContainerPort https://gateway.envoyproxy.io/docs/api/extension_types/#envoyproxykubernetesprovider
https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/

1 reply

finomen Apr 11, 2025
Author

I don't care about containerPort. But as LoadBalancer created it routes 443 port to nowhere

What I want is to route external(listener) port 443 internal service ports

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Unable to get gateway to listen on port 80 (containerPorts are not updated) #5707

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Unable to get gateway to listen on port 80 (containerPorts are not updated) #5707

Uh oh!

finomen Apr 10, 2025

Replies: 1 comment · 1 reply

Uh oh!

arkodg Apr 11, 2025 Maintainer

Uh oh!

Uh oh!

finomen Apr 11, 2025 Author

finomen
Apr 10, 2025

Replies: 1 comment 1 reply

arkodg
Apr 11, 2025
Maintainer

finomen Apr 11, 2025
Author