Skupper `skupper-network-status` ConfigMap Causing Significant ETCD Growth

[vsomwanshi]

Based on above architecture current deployment topology explained:

The production environment consists of:

• 1 Skupper Hub Site deployed on the ROKS cluster.
• Approximately 76 Skupper Spoke Sites deployed on ROSA clusters.
• Each spoke site is connected to the hub using a Skupper token generated from the hub site.
• Every ROSA namespace exposes approximately 17 services to the hub site.

As a result:

• Total exposed services = 76 × 17 = ~1,292 services
• All service definitions and network topology information are aggregated and maintained by the hub site.

Observed Issue:

The skupper-network-status ConfigMap in the hub namespace has grown to approximately 845 KB and continues to increase as:

• New namespaces are added and linked.
• Additional services are exposed.
• Site connectivity changes occur.
• Router and topology status updates are propagated across the network.

The ConfigMap is continuously synchronized across the Skupper network, generating frequent updates and creating significant pressure on ETCD storage.

Why the ConfigMap Has Become a Problem :

1. Excessive ConfigMap Size

The skupper-network-status ConfigMap has reached approximately 845 KB, which is considerably larger than a typical ConfigMap (usually in the range of a few KB).

2. Large Replicated Footprint

Since the network status is synchronized across all participating sites:

ConfigMap size: 845 KB
Number of connected sites: 76

This results in roughly:

845 KB × 76 ≈ 64.2 MB

of replicated configuration data across the environment.

3. High Update Frequency

The ConfigMap is updated whenever there are changes such as:

• Services being exposed or unexposed
• Site links connecting or disconnecting
• Router status changes
• Network topology updates
• Namespace onboarding activities

Given the scale of the deployment, these updates occur frequently.

4. ETCD Revision Amplification

Every ConfigMap modification creates a new revision in ETCD.

Assuming the full 845 KB object is rewritten per update:

Updates Approximate ETCD Consumption
10 ~642 MB
50 ~3.2 GB
100 ~6.4 GB

At the current scale, the skupper-network-status ConfigMap becomes a major contributor to ETCD database growth and fragmentation. It results in substantial ETCD bloat. This is contributing significantly toward ETCD database growth beyond the recommended operational thresholds and is a likely driver behind the observed ETCD usage spikes approaching or exceeding 8 GB.

How To Reproduce:

Not sure whether you could be able to reproduce it in your environment.

Expected behavior:

N/A

Environment details:

• Skupper CLI: 2.1.0
• Skupper Operator (if applicable): N/A
• Platform: 4.18.x

[c-kruse]

Hi @vsomwanshi, thank you for the issue.

This is behaving largely as we would expect given your configuration. We are just beginning to more seriously look for solutions that remove the skupper-network-status ConfigMap, due to these limitations and others. Presently this configmap plays an important role in how the skupper control plane works: there are no levers for you to pull to turn it off, compress its size or otherwise reduce the poor scaling behavior you are seeing.

In the short term, from what I understand you may be able to get quick wins reducing the effects of this behavior if you are able to partition your network. This would reduce both the size of these ConfigMaps and the write op frequency.

Will discuss further tomorrow!

[vsomwanshi]

@c-kruse As you suggested to explore on AttachedConnector instead of having connector in each site. I found one case whre it is mentioned that where multiple applications want to have a shared resources (e.g. postgres datbase), Skupper has the concept of an attached connector where the shared resource can be securely accessed by multiple VANs.

In our case we are not having shared services. We have 17 services in each site ( ~76 sites ) which needs to be exposed in other cluster so if in case we go for Attached connectors then we would need to create ~1,292 Attached connectors and bound them in the respective sites so in each namespace i would have to create 17 Attachedconnectorbindings. Is this solution would be feasible ? would it solve our motive to reduce the consumption of skupper-network-status configmap ?

[SidBB]

Having discussed this a bit offline with @vsomwanshi and reading through some docs, let me try to summarize and see if I understand the proposed changes.

@c-kruse , does the following sound right, and are you saying that this proposal would help reduce the size of the configmap?

Our current architecture:

• A "hub" namespace in cluster1
• 76 "spoke" namespaces in cluster2
• The hub needs to be able to connect to all 76 spokes
• Each of the 76 spoke namespaces has 17 services exposed
• Currently each of the 76 namespaces is defined as its own site = 77 sites in total
• Each of the 76 namespaces has 17 connectors defined in each site = 1292 connectors total in cluster2

Proposed improved architecture:

• Keep existing hub namespace as-is in cluster1
• Define a single "gateway" namespace in cluster2 and designate it as a skupper site
• Only 2 sites in the network, one on each cluster
• In each of the 76 namespaces in cluster2, define 17 AttachedConnectors = 1292 AttachedConnectors total
• In the gateway namespace in cluster2, define 1292 AttachedConnectorBindings

So there would be no change in the number of connectors / attachedconnectors overall, but it would reduce the number of sites in the network.

[c-kruse]

@SidBB could not have said it better myself. That is exactly what we are proposing.

[SidBB]

I'm trying out a small POC of this proposal using some mock services in a couple of single-node k3s clusters for testing, and I'm having trouble getting the AttachedConnectors to work. The status of the Binding reports "Error" and says it can't find a matching AttachedConnector. I'm using skupper v2.2.1.

I'd appreciate some help figuring out how to troubleshoot further.

Quick summary of my environment. On cluster1, I've deployed namespace-scoped skupper alongside my sample "ui" app:

cluster1> kubectl get pod -A

NAMESPACE     NAME                                        READY   STATUS      RESTARTS      AGE
isd-core      skupper-controller-65b95df5dc-mhkqm         1/1     Running     0             65m
isd-core      skupper-router-75877fd48f-2nthr             2/2     Running     0             58m
isd-core      ui-74776cf964-89qrz                         1/1     Running     2 (16h ago)   25h

On cluster 2, I have a few different "inventory-manager" apps in different namespaces. I want my ui app to be able to call all these inventory managers. I've deployed namespace-scoped skupper in a "gateway" namespace as discussed earlier in this thread.

cluster2> kubectl get pod -A
NAMESPACE         NAME                                        READY   STATUS      RESTARTS      AGE
isd-pl-11111      inventory-manager-8dc8755c8-2fs7n           1/1     Running     2 (16h ago)   25h
isd-pl-22222      inventory-manager-8dc8755c8-sgq74           1/1     Running     2 (16h ago)   25h
isd-pl-33333      inventory-manager-8dc8755c8-wvhgk           1/1     Running     2 (16h ago)   25h
skupper-gateway   skupper-controller-65b95df5dc-n5s7n         1/1     Running     0             65m
skupper-gateway   skupper-router-5b894778bf-hwwv2             2/2     Running     0             59m

I created and linked the two sites as seen here:

cluster1> kubectl get site -A
NAMESPACE   NAME        STATUS   SITES IN NETWORK   MESSAGE
isd-core    core-site   Ready    2                  OK

cluster2> kubectl get site -A
NAMESPACE         NAME       STATUS   SITES IN NETWORK   MESSAGE
skupper-gateway   den-site   Ready    2                  OK

cluster2> kubectl get link -A
NAMESPACE         NAME                                             STATUS   REMOTE SITE   MESSAGE
skupper-gateway   core-site-dce726de-3651-4989-b557-ca1503ca769a   Ready    core-site     OK

To start with, I've defined a single listener, attachedconnector, and attachedconnectorbinding using the examples in this repo. Here's how I defined them:

# Listener on cluster1
apiVersion: skupper.io/v2alpha1
kind: Listener
metadata:
  name: inventory-manager-11111
  namespace: isd-core
spec:
  host: inventory-manager-11111
  port: 8080
  routingKey: inventory-manager-11111
  type: tcp
---
# AttachedConnector & Binding on cluster2
apiVersion: skupper.io/v2alpha1
kind: AttachedConnector
metadata:
  name: inventory-manager-11111
  namespace: isd-pl-11111
spec:
  port: 8080
  selector: app=inventory-manager
  siteNamespace: skupper-gateway
  type: tcp
---
apiVersion: skupper.io/v2alpha1
kind: AttachedConnectorBinding
metadata:
  name: inventory-manager-11111
  namespace: skupper-gateway
spec:
  connectorNamespace: isd-pl-11111
  routingKey: inventory-manager-11111
---

But the binding reports an "Error" status:

cluster1> kubectl -n isd-core get listener
NAME                      ROUTING KEY               PORT   HOST                      STATUS    HAS MATCHING CONNECTOR   MESSAGE
inventory-manager-11111   inventory-manager-11111   8080   inventory-manager-11111   Pending                            No matching connectors

cluster2> kubectl -n isd-pl-11111 get attachedconnector
NAME                      PORT   SELECTOR                SITE NAMESPACE    STATUS
inventory-manager-11111   8080   app=inventory-manager   skupper-gateway

cluster2> kubectl -n skupper-gateway get attachedconnectorbinding
NAME                      ROUTING KEY               CONNECTOR NAMESPACE   STATUS   HAS MATCHING LISTENER
inventory-manager-11111   inventory-manager-11111   isd-pl-11111          Error

The detailed status of the binding reports it can't find a matching attachedconnector:

status:
conditions:
- lastTransitionTime: "2026-07-05T16:04:55Z"
  message: No matching AttachedConnector
  observedGeneration: 1
  reason: Error
  status: "False"
  type: Configured
- lastTransitionTime: "2026-07-05T16:04:55Z"
  message: No matching AttachedConnector
  observedGeneration: 1
  reason: Error
  status: "False"
  type: Ready
message: No matching AttachedConnector
status: Error

Edited to add: I don't see anything helpful in the logs for the skupper-controller or skupper-router pods. Last activity in there is from the time when I created the sites/links. Nothing logged when I created these listeners/connectors.

[c-kruse]

@SidBB I've got a few ideas. Is there any chance that cluster2 in your example is using a namespace-scoped skupper controller installation? I think we only permission it for the namespace it is running in (e.g. skupper-gateway in your case.) My other idea is that it could be the case that your selector app=inventory-manager does not match anything in isd-pl-11111? kubectl get pods -n isd-pl-11111 -l app=inventory-manager? I think we may also respect pod readiness - which may be worth checking on too. I was just about able to reproduce this with a bad/empty selector (but saw status Error on the attachedconnector, so I'm confident that's what this is.)

[SidBB]

@c-kruse, you're right, it was the namespace-scoped controller, and that does make sense that its permissions would be limited. I reconfigured with a cluster-scoped controller on both sides and got everything working. Thanks for the quick response! This was just a small test but we'll try it out in a large-sized pre-prod environment in the coming weeks.

At least for the small env, I compared the contents of the configmap skupper-network-status, and I do see the benefits of using the "gateway" site because it cuts down on some of the redundant info that is nested under each site. However, it doesn't impact the number of connectors that are tracked in the configmap overall. Instead of being nested under different sites, they would all be nested under one site, but the total number would still be the same. So in the future, the configmap might still grow as we add more namespaces and more services. Anyway, at least this should address the pressure in the near term. Using some rough calculations, I estimate that for our prod environnent of 77 sites, we would save about 400KB in the configmap if we reduce it down to 2 sites.

One other question: should we be concerned about the performance of the router in the "gateway" namespace if it's handling combined traffic that was previously being handled by 76 different routers? Then again, we do have a single router on the "hub" cluster, so if that can handle the combined calls to 76 "spoke" sites, I suppose that's proof enough that it can handle it. But is there an option to scale the router if the need comes up in the future?

[c-kruse]

@SidBB we are in agreement on the importance of addressing the underlying issue with skupper's ConfigMap utilization moving forward.

I believe that by not (effectively) duplicating the same skupper-network-status ConfigMap across your 77 Sites all competing for the same cluster/etcd resources you should save much more than 400KB. One warning, as you go through evaluating this single site approach: depending on many factors (routing key naming scheme, number of pods targeted, cluster IP addressing v4 or v6), you may find that the skupper-router ConfigMap gets close to exceeding the 1MB limit as you add that many (Attached)Connectors to the same Site. I'm afraid you may end up with a balancing act finding a comfortable number of "gateway" Sites for your application.

One other question: should we be concerned about the performance of the router in the "gateway" namespace if it's handling combined traffic that was previously being handled by 76 different routers? Then again, we do have a single router on the "hub" cluster, so if that can handle the combined calls to 76 "spoke" sites, I suppose that's proof enough that it can handle it.

That is pretty much how I'd think about it as well: I would begin with resourcing the new Site with the same compute resources as the "hub" cluster.

But is there an option to scale the router if the need comes up in the future?

There's a few ways to go about it:

• more cpu and memory resources for the router.
• You may observe some benefits enabling HA (deploys two router deployments/two pods) depending on your workload.
• You could add a Site per-N application namespaces. I'm not sure that'd yield much performance/throughput benefit without first leaving the "hub" Site approach behind, though.