Use karmada-search for cache resource

[JimDevil]

pprof file:pprof.zip

Please provide an in-depth description of the question you have:
We want to use the resourceregistry CR of karmada-search to cache resources in order to avoid the issue of the apiserver crashing due to a large number of informer list requests.

So, we conducted a performance test，and here is the test situation.

Testing Environment

2 k8s cluster. one is admin cluster,another one is member cluster

Test case

I want to simulate a list request that responds with a large number of resources at once.

1. Create 200 secrets in the member cluster, with each secret being 1M in size, expecting a single list to return 200M of data.
2. Cache the secrets of the member cluster through the ResourceRegistry.

after normal startup and resource cached

Memory usage is around 600M.

start 1 informer

alloc_space

cpu

memory usage

The maximum value reached 1.8G.

start 5 informer

alloc_space

cpu

memory usage

The maximum value reached 5.86G.

start 10 informer

OOM, karmada-seach pod restarts because I set the limit to 8G,message from describe pod

Reason:       OOMKilled

What do you think about this question?:
I think the use of memory seems a bit abnormal.I assumed that one informer request would consume around 200M of memory, but in the program, the actual allocation exceeded 1G.

At the same time, two problems were found.

1. From the memory profile,it can be seen that memory growth is mainly due to byte.growSlice.
2. From the cpu profile,it can be seen that runtime.memclrNoheapPointers takes a longer time.

k8s official solution for this situation is to enable watchlist, but it requires a higher version.
We want to use the karmada-search solution in a lower-version cluster.
Are there any optimization solutions?

Environment:

• Karmada version: 1.12
• Kubernetes version:
• Others:

[JimDevil]

@RainbowMango

[RainbowMango]

I think the use of memory seems a bit abnormal.I assumed that one informer request would consume around 200M of memory, but in the program, the actual allocation exceeded 1G.

Yeah, I'm surprised too.

At the same time, two problems were found.
From the memory profile,it can be seen that memory growth is mainly due to byte.growSlice.
From the cpu profile,it can be seen that runtime.memclrNoheapPointers takes a longer time.
k8s official solution for this situation is to enable watchlist, but it requires a higher version.

Do you have any link to the k8s official solution?

I need time to figure out the current situation and explore potential solutions.

By the way, would you like to share this test report at the community meeting? Or the SIG-Scalability meeting, which focuses on performance enhancement.

[JimDevil]

I think the use of memory seems a bit abnormal.I assumed that one informer request would consume around 200M of memory, but in the program, the actual allocation exceeded 1G.

Yeah, I'm surprised too.

At the same time, two problems were found.
From the memory profile,it can be seen that memory growth is mainly due to byte.growSlice.
From the cpu profile,it can be seen that runtime.memclrNoheapPointers takes a longer time.
k8s official solution for this situation is to enable watchlist, but it requires a higher version.

Do you have any link to the k8s official solution?

https://kubernetes.io/zh-cn/blog/2024/12/17/kube-apiserver-api-streaming/

I need time to figure out the current situation and explore potential solutions.

By the way, would you like to share this test report at the community meeting? Or the SIG-Scalability meeting, which focuses on performance enhancement.

[RainbowMango]

Thanks, I will take a look and get back to you.

[JimDevil]

I think the use of memory seems a bit abnormal.I assumed that one informer request would consume around 200M of memory, but in the program, the actual allocation exceeded 1G.

Yeah, I'm surprised too.

At the same time, two problems were found.
From the memory profile,it can be seen that memory growth is mainly due to byte.growSlice.
From the cpu profile,it can be seen that runtime.memclrNoheapPointers takes a longer time.
k8s official solution for this situation is to enable watchlist, but it requires a higher version.

Do you have any link to the k8s official solution?

I need time to figure out the current situation and explore potential solutions.

By the way, would you like to share this test report at the community meeting? Or the SIG-Scalability meeting, which focuses on performance enhancement.

This test report is not complicated，I don't think there is much to share about this test report. I am looking for a way to solve the problem. Maybe it would be better to share after the problem is solved.

[RainbowMango]

OK. I will try my best to explore it.
try to ping karmada-search author @ikaven1024 for any hints

[JimDevil]

@RainbowMango
I have consulted some materials. The problem of this memory consumption is probably due to the following reasons:

1. kube-apiserver needs to perform DeepCopy and serialization operations on data.
2. The DeepCopy operation will generate temporary memory.
3. Serialization also requires a buffer to store the converted byte stream.

Therefore, a list request in kube-apiserver often requires several times the memory of the actual response data.
WatchList can solve this problem，But in lower versions that do not support the watchlist feature, can it be optimized through other methods?

Can karamda-search support removing unnecessary attributes in the resource object like the SetTransform method?

There is a common attribute "managedFields" on all resource objects, which exists in ObjectMeta. This attribute can be ignored in almost all client code that uses Informer.

Can an enableManagedFields field be added to ResourceRegistry, with a default value of false, so that the content of the ManagedFields field is not stored in the cache of karmada-search to save some memory? Similarly, some memory consumption can be saved in deep copying and serialization operations.

For example

apiVersion: search.karmada.io/v1alpha1
kind: ResourceRegistry
metadata:
  name: proxy-sample
spec:
  targetCluster:
    clusterNames:
  resourceSelectors:
    - apiVersion: v1
      kind: Pod
      enableManagedFields: false

[RainbowMango]

Can karamda-search support removing unnecessary attributes in the resource object like the SetTransform method?

I think yes. And have removed .metadata.managedFields, see:

karmada/pkg/search/controller.go

Lines 125 to 127 in 7f7879d

	// ignore the error here because the informers haven't been started
	_ = clusterInformer.SetTransform(fedinformer.StripUnusedFields)
	_ = resourceRegistryInformer.SetTransform(fedinformer.StripUnusedFields)

[JimDevil]

Can karamda-search support removing unnecessary attributes in the resource object like the SetTransform method?

I think yes. And have removed .metadata.managedFields, see:

karmada/pkg/search/controller.go

Lines 125 to 127 in 7f7879d

// ignore the error here because the informers haven't been started
_ = clusterInformer.SetTransform(fedinformer.StripUnusedFields)
_ = resourceRegistryInformer.SetTransform(fedinformer.StripUnusedFields)

Oh,this seems to only remove the.metadata.managedFields of the cluster and resourceRegistry resources in the search module. What I had in mind was that the proxy module could remove the.metadata.managedFields of the resources GVK configured in the resourceRegistry, such as pods、secrets.
Just as I said above

[RainbowMango]

Do you mean to introduce another field to the ResourceRegistry API?
How about having a chat at one of the community meetings?