bug: APIExportEndpointSlice only exposes internal shard URL which causes api-syncagent to crash

[Rand0mF]

Describe the bug

The sync agent tries to do some calls to an URL from the remote APIExportEndpointSlice.status.Endpoints.URL. However, the URL written there is kcp cluster internal and not the URL of the front-proxy, so the call obviously fails from the external sync-agent cluster and the sync-agent pod is crashing.

{"level":"fatal","time":"2026-02-06T13:38:52.993Z","logger":"syncagent-syncmanager","caller":"syncmanager/controller.go:332","msg":"Failed to start apiexport provider","error":"failed to get *v1alpha1.APIBinding informer: failed to get server groups: Get \"https://kcp:6443/services/apiexport/4flkw2hcw8v04w7f/my.example.group/clusters/%2A/api\": dial tcp: lookup kcp on 34.118.224.10:53: no such host"}

The URL of the APIExportEndpointSlice is directly set from the kcp controller parameter --shard-base-url (relevant code line#1 and #2). This URL is not the front-proxy url.
The parameter is configured here in the helm values and it differs from the external URL, which is defined one line below.

Steps To Reproduce

1. install kcp on a cluster with the helm chart and values:

externalHostname: my.external.hostname
kcpFrontProxy:
  service:
    type: LoadBalancer

configure the externalHostname DNS record to point to the front-proxy loadbalancer.
Now KCP works normally.

2. Add an (empty) APIExport, which will later be referenced by the api-syncagent

3. install the kcp api-syncagent on a second cluster and point it to the APIExport on KCP.

4. Create a Binding on KCP for the APIExport.

Now you will notice, the APIExportEndpointSlice gets filled with a URL: (removed some fields for brevity)

Name:         my.example.group
Annotations:  kcp.io/cluster: 4flkw2hcw8v04w7f
API Version:  apis.kcp.io/v1alpha1
Kind:         APIExportEndpointSlice
Spec:
  Export:
    Name:  my.example.group
    Path:  root:ws-1:ws-2
Status:
  Conditions:
    Last Transition Time:  2026-02-06T13:31:24Z
    Status:                True
    Type:                  APIExportValid
    Last Transition Time:  2026-02-06T13:31:24Z
    Status:                True
    Type:                  PartitionValid
  Endpoints:
    URL:  https://kcp:6443/services/apiexport/4flkw2hcw8v04w7f/my.example.group
Events:   <none>

The sync-agent crashes now because it tries to call this URL which is obviously not reachable by the external cluster.

Expected Behaviour

I would expect the APIExportEndpointSlice to (additionally) contain the external (front-proxy) URL, which is then uses by the api-syncagent and it doesn't crash.
I evaluated it with this PR and can confirm that it works.

Additional Context

No response

[ntnn]

Hi @Rand0mF!

As mentioned in #3823 the front proxy is not supposed to route virtual workspaces and that just happens to work for the root shard - but that is not intended.

A shard is configured with three URLs:

1. external address -> the address the front proxy is reachable under
2. virtual workspace url -> the address of the virtual workspace server for that shard to generate virtual workspace URLs
3. shard base url -> the address of the shard itself

In a two shard setup it can look like this:

1. front-proxy: https://api.kcp.example.com/
2. root shard base: https://root.api.kcp.example.com/
3. root vw base: https://root.api.kcp.example.com/ (so same as the shard base)
4. second shard base: https://second.api.kcp.example.com/
5. second vw base https://vw.second.api.kcp.example.com/ (so different from the shard base)

If the VW URL would be generated off of the front-proxy url the virtual workspace URL for both the root and the second shard would be https://api.kcp.example.com/. So there would be no way for the front-proxy to know which of the two shards you are targeting.

For this reason you need to expose the virtual workspace endpoints of your shards - which is the same as the shard base url unless you run the virtual workspace separately from the shard instance.

The front-proxy is meant as an easy entrance for clients to the clusters. This proxies the requests to the correct shard. Virtual workspaces, which mostly run alongside shard, are meant to be accessed directly. Shards are also generally recommended to be accessed directly when possible to lessen the load on the front-proxy.

[ntnn]

You can setup an ingress and set the shard base url with --shard-base-url in the extraFlags for the kcp instance: https://github.com/kcp-dev/helm-charts/blob/35cafc6ca140fe4464cbcd9a5e3c9c0bebd6b88e/charts/kcp/values.yaml#L85

Alternatively for a single front-proxy/single shard setup you can also pass the front proxy url as the virtual workspace url of the shard with --virtual-workspace-base-url - as mentioned it incidentally works that the virtual workspace of the root shard is accessible through the front proxy but that is not intended.

[Rand0mF]

@ntnn I tried your suggestion and it does not work together with the helm chart.
The issue is when specifying --shard-base-url as extraFlags this flag will be templated 2 times from the helm chart. (first time here and second time here) I think the first argument wins.

Additionally, even if it would fix it, I think it would require updated guides for the api-syncagent. As this behavior is not intuitive at all, esp. that it crashes per default.

I think everyone who follows the KCP installation guide via the helm chart (on a real cluster) and then installs the api-syncagent will run in the same issue.

edit: forgot to test --shard-virtual-workspace-url, with this one set it works. But I still think the usage is not very intuitive (or also unsure if it's a good idea to overwrite the virtual workspace url when there is a proper front-proxy installation running 🤔)

[SimonTheLeg]

A shard is configured with three URLs:

1. external address -> the address the front proxy is reachable under
2. virtual workspace url -> the address of the virtual workspace server for that shard to generate virtual workspace URLs
3. shard base url -> the address of the shard itself

@ntnn I think there is a fourth one, --shard-external-url. Do you know why we differentiate shards with base and external url, but vw only with one? (--shard-virtual-workspace-url). I would assume that there is a case where you would want to run some of the consumers of VW to run inside the same cluster as kcp and some to run outside.

Additionally I am wondering if we should make this whole "VW consumers are running outside the cluster that kcp is running in" as a supported case in our helm chart. "Essentially if .Values.shardVirtualWorkspaceURL is set, then we create another ingress/GatewayAPI route? This is after all at least the case for everyone using the api-syncagent, I think.

[ntnn]

@ntnn I tried your suggestion and it does not work together with the helm chart. The issue is when specifying --shard-base-url as extraFlags this flag will be templated 2 times from the helm chart. (first time here and second time here) I think the first argument wins.

Additionally, even if it would fix it, I think it would require updated guides for the api-syncagent. As this behavior is not intuitive at all, esp. that it crashes per default.

I think everyone who follows the KCP installation guide via the helm chart (on a real cluster) and then installs the api-syncagent will run in the same issue.

edit: forgot to test --shard-virtual-workspace-url, with this one set it works. But I still think the usage is not very intuitive (or also unsure if it's a good idea to overwrite the virtual workspace url when there is a proper front-proxy installation running 🤔)

Yeah that is true - I had a quick look over the helm chart docs and it does seem outdated. We mostly use the kcp-operator now: https://docs.kcp.io/kcp-operator/
Nonetheless we should update the helm chart docs accordingly to expose both the shards and the front-proxy (and template --shard-base-url so it can be configured).

A shard is configured with three URLs:

1. external address -> the address the front proxy is reachable under
2. virtual workspace url -> the address of the virtual workspace server for that shard to generate virtual workspace URLs
3. shard base url -> the address of the shard itself

@ntnn I think there is a fourth one, --shard-external-url. Do you know why we differentiate shards with base and external url, but vw only with one? (--shard-virtual-workspace-url). I would assume that there is a case where you would want to run some of the consumers of VW to run inside the same cluster as kcp and some to run outside.

Additionally I am wondering if we should make this whole VW consumers are running outside the cluster that kcp is running in as a supported case in our helm chart. "Essentially if .Values.shardVirtualWorkspaceURL is set, then we create another ingress/GatewayAPI route?

--shard-external-url is the front-proxy address and what shows up in the shards object; the naming is a bit all over the place which I'd attribute to the growing pains from when the architecture wasn't set in stone yet (--shard-external-url and --external-hostname should be the same, but --external-hostname comes from kube).

The difference is that --shard-base-url is the url to reach the shard directly e.g. with kube clients that talk to logical clusters on that shard and for the front-proxy to talk to the shard. --shard-external-url is for the front-proxy as mentioned. VW doesn't have this distinction because the VW only exists per-shard next to the shard (or as a global VW mapped in the front-proxy by the user) and isn't accessible through the front-proxy.

The --shard-virtual-workspace-url is mostly so users can run the shard and its respective virtual workspace separately if they wish.
The helm charts are complex enough imho - we should only add adding an ingress for the root shard and set the shard base url accordingly, similar like with the front-proxy.

[ntnn]

I had a quick look and we override the --external-hostname anyhow when it is not set explicitly:

kcp/pkg/server/options/options.go

Lines 292 to 306 in e6c6375

	// ExternalAddress is the address used e.g. when generating
	// kubeconfigs. It defaults to the default interface, usually the
	// first non-loopback interface, e.g. 192.168.0.1.
	// BindAddress is the address the server binds to, it defaults to
	// 0.0.0.0 or ::.
	//
	// If BindAddress is set to a specific address, e.g. the loopback
	// 127.0.0.1 the ExternalAddress is invalid and all URLs generated
	// from it will not work.
	//
	// To prevent this ExternalAddress is set to the value of
	// BindAddress if it wasn't set to a specific address.
	if o.GenericControlPlane.GenericServerRunOptions.ExternalHost == "" && !o.GenericControlPlane.SecureServing.BindAddress.IsUnspecified() {
	o.GenericControlPlane.GenericServerRunOptions.ExternalHost = o.GenericControlPlane.SecureServing.BindAddress.String()
	}

I think it would make sense to set this to the host in the shard base url and disallow the flag. I'll try that and see what breaks.