What happened?
On a Gardener cluster
kubectl logs, kubectl exec, kubectl attach, and kubectl port-forward all return 404 when vcluster is running on a host cluster whose kube-apiserver is fronted by an istio-envoy proxy.
Regular API calls (get pods, get nodes, etc.) work fine.
Error from server (NotFound): the server could not find the requested resource (get pods coredns-76cc888f5-hfm8q)
The syncer logs show nothing — even with DEBUG=true, no proxy or redirect entry appears for the failing request. The 404 comes back silently.
What did you expect to happen?
Pod SubResource operations should work regardless of whether the host kube-apiserver is behind an envoy proxy.
How can we reproduce it (as minimally and precisely as possible)?
- Deploy vcluster on a Gardener shoot cluster (or any host cluster with istio-envoy in front of the kube-apiserver)
kubectl get pods -A → workskubectl logs <any-pod> → Error from server (NotFound)
Anything else we need to know?
Root cause
From inside the syncer container, a plain GET to the host apiserver reveals it is behind istio-envoy:
$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ wget -q -S -O- --no-check-certificate \
--header="Authorization: Bearer $TOKEN" \
"https://api.<host-cluster-domain>/api/v1/namespaces/<ns>/pods" 2>&1 | head -5
HTTP/1.1 200 OK
server: istio-envoy
x-envoy-upstream-service-time: 5Istio-envoy routes by Host header. Testing the logs subresource directly with and without the correct Host header isolates the problem:
# correct Host header → istio routes to apiserver, returns 200 with log content
wget --no-check-certificate -S -O- \
--header="Authorization: Bearer $TOKEN" \
"https://api.<host-cluster-domain>/api/v1/namespaces/<ns>/pods/<pod>/log?limitBytes=200"
# HTTP/1.1 200 OK
# server: istio-envoy
# x-envoy-upstream-service-time: 39
# [log content follows]
# vcluster's actual Host header (client-facing vcluster hostname) → istio returns 404
wget --no-check-certificate --header="Authorization: Bearer $TOKEN" \
--header="Host: myvcluster.<public-domain>" \
"https://api.<host-cluster-domain>/api/v1/namespaces/<ns>/pods/<pod>/log"
# HTTP/1.1 404 Not Found
When vcluster proxies pod subresource requests to the host cluster the outgoing request carries the original client-facing Host header instead of the host apiserver's hostname. Istio-envoy has no virtual host entry for the vcluster's public name and returns 404.
Host cluster Kubernetes version
Details
$ kubectl version
Client Version: v1.35.0
Kustomize Version: v5.7.1
Server Version: v1.34.5
vcluster version
Details
$ vcluster --version
0.34.0
VCluster Config
Deployed via helm
Details
controlPlane:
distro:
k8s:
enabled: true
image:
tag: "v1.34.5"
apiServer:
enabled: true
controllerManager:
enabled: true
scheduler:
enabled: true
backingStore:
etcd:
deploy:
enabled: true
statefulSet:
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
persistence:
volumeClaim:
size: 1Gi
retentionPolicy: Delete
statefulSet:
env:
- name: DEBUG
value: "true"
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
cpu: 1000m
memory: 2Gi
persistence:
addVolumes:
- name: auth-config
configMap:
name: auth-config
addVolumeMounts:
- name: auth-config
mountPath: /etc/kubernetes/auth
readOnly: true
security:
podSecurityContext:
fsGroup: 12345
containerSecurityContext:
runAsUser: 12345
runAsNonRoot: true
sync:
toHost:
ingresses:
enabled: true
pods:
enabled: true
persistentVolumes:
enabled: true
persistentVolumeClaims:
enabled: true
fromHost:
nodes:
enabled: true
selector:
all: true
storageClasses:
enabled: true
selector:
matchExpressions:
- key: resources.gardener.cloud/managed-by
operator: In
values:
- gardener
policies:
resourceQuota:
enabled: false
limitRange:
enabled: false
networkPolicy:
enabled: false
What happened?
On a Gardener cluster
kubectl logs,kubectl exec,kubectl attach, andkubectl port-forwardall return 404 when vcluster is running on a host cluster whosekube-apiserveris fronted by anistio-envoy proxy.Regular API calls (
get pods,get nodes, etc.) work fine.The syncer logs show nothing — even with
DEBUG=true, no proxy or redirect entry appears for the failing request. The 404 comes back silently.What did you expect to happen?
Pod SubResource operations should work regardless of whether the host
kube-apiserveris behind an envoy proxy.How can we reproduce it (as minimally and precisely as possible)?
kubectl get pods -A→ workskubectl logs <any-pod>→Error from server (NotFound)Anything else we need to know?
Root cause
From inside the syncer container, a plain GET to the host apiserver reveals it is behind istio-envoy:
Istio-envoy routes by
Hostheader. Testing the logs subresource directly with and without the correctHostheader isolates the problem:When vcluster proxies pod subresource requests to the host cluster the outgoing request carries the original client-facing
Hostheader instead of the host apiserver's hostname. Istio-envoy has no virtual host entry for the vcluster's public name and returns 404.Host cluster Kubernetes version
Details
vcluster version
Details
VCluster Config
Deployed via helm
Details