NetworkEgress Service omits targetPort — clients behind Istio ambient get connection reset

[raulmartinezr]

TL;DR: The Service created for a NetworkEgress sets targetPort equal to port instead of the allocated router forward port. Standard Kubernetes dataplanes ignore this (the manual EndpointSlice port wins), but Istio ambient's ztunnel resolves the VIP using the Service targetPort, so every mesh-captured client gets connection reset by peer. Setting targetPort to the forward port fixes it (verified live).

Environment

• kubernetes-operator 0.8.0 (Helm chart 0.8.0), NetBird 0.74.6 self-hosted
• EKS 1.34, Cilium (kube-proxy replacement), Istio 1.28.6 ambient mesh

What happens

NetworkEgress reconciles fine (Ready=True) and creates:

• a selectorless Service: port: 9876, targetPort: 9876 (defaulted, since the controller never sets it)
• a manual EndpointSlice: router pod IP, port: 38035 (the forward port the router actually listens on)

The two objects contradict each other, and each dataplane believes a different one:

Client pod	direct podIP:38035	Service VIP :9876
ambient-captured (ztunnel)	200	connection reset after the request is sent
not captured (istio.io/dataplane-mode: none)	200	200

Cilium programs the VIP correctly from the slice (cilium service list shows VIP:9876 -> podIP:38035), which is why non-captured pods work: for selectorless Services the EndpointSlice port is authoritative and targetPort is ignored. ztunnel instead resolves the VIP with the Service's targetPort and dials podIP:9876, where nothing listens, so the TCP connection is reset.

In an ambient-mesh cluster this means no meshed workload can consume a NetworkEgress Service — which is most workloads in such clusters.

Reproduction

1. Cluster with Istio ambient; a namespace labeled istio.io/dataplane-mode: ambient.
2. Create a NetworkEgress in that namespace targeting any reachable peer FQDN/port.
3. curl http://<egress-service>:<port>/ from a pod in that namespace → Recv failure: Connection reset by peer.
4. Same curl from a pod labeled istio.io/dataplane-mode: none → works.

Fix (verified)

The controller builds the Service ports without a target port (networkegress_controller.go):

ports = append(ports, corev1ac.ServicePort().WithName(port.Name).WithPort(port.Port))

Adding .WithTargetPort(intstr.FromInt32(<allocated forward port>)) makes the Service agree with the EndpointSlice, and both resolution paths converge. Verified live: after kubectl patch svc <egress> --type=json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":38035}]', the same ambient-captured pod that got the reset receives HTTP 200 through the tunnel.

Arguably ztunnel should also honor manual EndpointSlice ports for selectorless Services (happy to file that on the Istio side too), but making the operator emit self-consistent objects fixes it for every dataplane today.

[linear-code]

NET-1423

[phillebaba]

This is a weird bug, I spent some time making sure that it was ok to set a different target port in the endpoint slice. The logistical reason is because the egress reconciler is responsible for creating the service while the egress service reconciler is responsible for allocating random ports and managing endpoint slices. It is split this way as we need to ensure that there is no overlap off random ports per network router, so they are generated from one reconcile loop.

Could you also create an issue in Istio to see if they consider this a bug in there end. It may take some time to get this fixed in Istio, so we may need to think of a solution in the meantime.

[raulmartinezr]

Thanks for the quick triage and the explanation of the two-reconciler split — that makes sense (the random-port allocation living in the egress-service reconciler is exactly why the Service can't trivially carry the targetPort at creation time).

Filed the Istio side as requested: istio/istio#60951 — it reports that ztunnel resolves a selectorless Service's VIP by Service.targetPort and ignores the manually-managed EndpointSlice port (kube-proxy / Cilium do the opposite for selectorless Services). Includes a standalone minimal repro that doesn't depend on this operator.

Agreed an Istio-side fix may take a while. For an interim, purely on the operator side: since the egress-service reconciler already owns the allocated forward port and writes it into the EndpointSlice, could it set the same value as the Service's targetPort when it (or a later reconcile pass) updates the Service? That single change makes the Service self-consistent with its slice and unblocks every dataplane today, independent of the Istio fix. Happy to test any patch against our ambient cluster (Istio 1.28.6 + Cilium) — a manual kubectl patch of targetPort to the forward port already flipped the meshed client from RST to 200 for us.

[raulmartinezr]

Correction to my earlier comment — please disregard the targetPort suggestion.

I ran a controlled test and my earlier diagnosis was wrong. Setting the Service targetPort to the forward port does not fix it (I verified: patching targetPort left ztunnel's endpoints empty and the connection still failed). targetPort is not involved.

The actual, isolated cause is on the Istio side: ztunnel populates no endpoints for a selectorless Service whose backends come from a manually-managed EndpointSlice, so its VIP is unroutable from ambient-meshed clients. Controlled repro, same backend pod and ports:

Service	ztunnel endpoints	curl from an ambient pod
with a selector (port 80 → targetPort 9000)	1	200
no selector + manual EndpointSlice (port 9000)	empty	fails

Filed on Istio with the full repro: istio/istio#60951 (I've rewritten it to reflect this; it no longer mentions targetPort).

For an operator-side interim that does not wait on Istio: if the egress Service used a selector targeting the routing-peer pod (with targetPort = the allocated forward port) instead of a selectorless Service + manual EndpointSlice, ambient routes it correctly (the "with a selector" row above). I realize that cuts against the current two-reconciler design where ports are allocated separately from Service creation, so it may not be trivial — flagging it as a direction, not a demand. Happy to test any approach against our ambient cluster (Istio 1.28.6 + Cilium).

Apologies for the noise from the first suggestion.