Skip to content

[k8sattributes] Add extended load test#47825

Open
ChrsMark wants to merge 1 commit intoopen-telemetry:mainfrom
ChrsMark:add_k8sattrr_extended_load_tests
Open

[k8sattributes] Add extended load test#47825
ChrsMark wants to merge 1 commit intoopen-telemetry:mainfrom
ChrsMark:add_k8sattrr_extended_load_tests

Conversation

@ChrsMark
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark commented Apr 21, 2026

Description

As described at #47389 (comment), we need an extended benchmark for the k8sattributes processor which will stress test the component not just against the default configuration but against an extended configuration which enables all the available workload type attributes (daemonset, statefulset etc).

This PR adds this extended test using the following configuration:

extract:
  metadata:
    - k8s.pod.name
    - k8s.pod.start_time
    - k8s.pod.uid
    - k8s.namespace.name
    - k8s.deployment.name
    - k8s.deployment.uid
    - k8s.replicaset.name
    - k8s.replicaset.uid
    - k8s.statefulset.name
    - k8s.statefulset.uid
    - k8s.daemonset.name
    - k8s.daemonset.uid
    - k8s.cronjob.name
    - k8s.cronjob.uid
    - k8s.job.name
    - k8s.job.uid
    - k8s.node.name
    - k8s.cluster.uid
    - container.image.name
    - container.image.tag
pod_association:
  - sources:
      - from: resource_attribute
        name: k8s.pod.uid

and for the following Workload types:

- Deployment → ReplicaSet → Pod
- StatefulSet → Pod
- DaemonSet   → Pod  (single-node cluster; 1:1 ratio)
- CronJob     → Job → Pod

The scenarios test against 110, 1K and 5K distinct workloads. This means the 1K scenario tests a kwok cluster with 1K Deployment, 1K Daemonsets, 1K Cronjobs and 1K StatefulSets so as to simulate a full scale K8s cluster with various types of workloads. This will allow us understand what is the impact in the memory footprint for the additional workload types that we add. Below the scenarios details:

// ── Test coverage reference ───────────────────────────────────────────────────
//
// # TestMetricK8sAttributesProcessor (basic)
//
// Cluster: 100 nodes, N namespaces
// Each Deployment owns exactly 1 ReplicaSet which owns exactly 1 Pod.
//
//	Scale  | Nodes | Namespaces | Deployments | ReplicaSets | Pods
//	-------|-------|------------|-------------|-------------|------
//	   110 |   100 |        110 |         110 |         110 |   110
//	 1 000 |   100 |      1 000 |       1 000 |       1 000 | 1 000
//	 5 000 |   100 |      5 000 |       5 000 |       5 000 | 5 000
//
// # TestMetricK8sAttributesProcessorExtended (extended)
//
// Cluster: 100 nodes, N namespaces in total with 4 namespaces being targeted for the workloads(1 per workload type).
// DaemonSet pods are pinned to a single node via nodeSelector (1 pod per DaemonSet).
//
//	Namespace        | Workload type | Ownership chain
//	-----------------|---------------|--------------------------------------
//	namespace-000000 | Deployment    | Deployment → ReplicaSet → Pod
//	namespace-000001 | StatefulSet   | StatefulSet → Pod
//	namespace-000002 | DaemonSet     | DaemonSet (nodeSelector) → Pod
//	namespace-000003 | CronJob       | CronJob → Job → Pod
//
//	Scale  | Nodes | NS | Deployments | RSes  | StatefulSets | DaemonSets | CronJobs | Jobs  | Total Pods
//	-------|-------|----|-------------|-------|--------------|------------|----------|-------|------------
//	   110 |   100 |  110 |         110 |   110 |          110 |        110 |      110 |   110 |        440
//	 1 000 |   100 |  110 |       1 000 | 1 000 |        1 000 |      1 000 |    1 000 | 1 000 |      4 000
//	 5 000 |   100 |  110 |       5 000 | 5 000 |        5 000 |      5 000 |    5 000 | 5 000 |     20 000

Link to tracking issue

Related to #47389

Testing

Running the benchmarks locally I get the following results:

--- PASS: TestMetricK8sAttributesProcessor (183.03s)
    --- PASS: TestMetricK8sAttributesProcessor/110_workload_cluster (23.56s)
    --- PASS: TestMetricK8sAttributesProcessor/1K_workload_cluster (40.15s)
    --- PASS: TestMetricK8sAttributesProcessor/5K_workload_cluster (119.32s)
...
--- PASS: TestMetricK8sAttributesProcessorExtended (291.94s)
    --- PASS: TestMetricK8sAttributesProcessorExtended/110_workload_cluster_extended (26.53s)
    --- PASS: TestMetricK8sAttributesProcessorExtended/1K_workload_cluster_extended (70.47s)
    --- PASS: TestMetricK8sAttributesProcessorExtended/5K_workload_cluster_extended (194.94s)
....
....
Test                                    |Result|Duration|CPU Avg%|CPU Max%|CPU Limit|RAM Avg MiB|RAM Max MiB|RAM Limit MiB|Sent Items|Received Items|
----------------------------------------|------|-------:|-------:|-------:|--------:|----------:|----------:|------------:|---------:|-------------:|
IdleMode                                |PASS  |     15s|     0.7|     2.9|     0.0|         52|         73|          0|         0|             0|
MetricK8sAttributesProcessor/110_workload_cluster|PASS  |     18s|     0.5|     2.1|     0.0|         57|         80|          0|        10|            10|
MetricK8sAttributesProcessor/1K_workload_cluster|PASS  |     19s|     0.8|     3.2|     0.0|         73|         99|          0|        10|            10|
MetricK8sAttributesProcessor/5K_workload_cluster|PASS  |     22s|     1.2|     7.7|     0.0|        142|        184|          0|        10|            10|
MetricK8sAttributesProcessorExtended/110_workload_cluster_extended|PASS  |     20s|     0.8|     3.5|     0.0|         67|         91|          0|        40|            40|
MetricK8sAttributesProcessorExtended/1K_workload_cluster_extended|PASS  |     24s|     1.7|    10.0|     0.0|        125|        161|          0|        40|            40|
MetricK8sAttributesProcessorExtended/5K_workload_cluster_extended|PASS  |     40s|    10.4|    37.3|     0.0|        507|        602|          0|        40|            40|

Results align with the current behaviour: https://open-telemetry.github.io/opentelemetry-collector-contrib/benchmarks/loadtests/#metrick8sattributesprocessor-5k-workload-cluster-ram-mib

@ChrsMark ChrsMark requested a review from a team as a code owner April 21, 2026 15:22
@ChrsMark ChrsMark requested a review from MovieStoreGuy April 21, 2026 15:22
@ChrsMark ChrsMark self-assigned this Apr 21, 2026
Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark force-pushed the add_k8sattrr_extended_load_tests branch from 22f45ff to f1e0402 Compare April 21, 2026 15:23
@ChrsMark ChrsMark added chore Skip Changelog PRs that do not require a CHANGELOG.md entry processor/k8sattributes k8s Attributes processor labels Apr 21, 2026
Copy link
Copy Markdown
Contributor

@MovieStoreGuy MovieStoreGuy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a few lint issues to address, but otherwise his looks reasonable?

Comment on lines -28 to -35
func skipIfKwokUnavailable(t *testing.T) {
if os.Getenv("SKIP_KWOK_TESTS") == "1" {
t.Skip("Skipping KWOK test: SKIP_KWOK_TESTS=1")
}
if _, err := exec.LookPath("kwokctl"); err != nil {
t.Skipf("Skipping KWOK test: kwokctl not found in PATH (install from https://kwok.sigs.k8s.io/)")
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected be removed from the changes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore processor/k8sattributes k8s Attributes processor Skip Changelog PRs that do not require a CHANGELOG.md entry testbed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants