Add tests for EBS CSI NVMe on EKS by duhminick · Pull Request #507 · aws/amazon-cloudwatch-agent-test

duhminick · 2025-04-17T18:39:02Z

Description of the issue

Testing that the metrics and EMF logs are correct when CWA is emitting EBS NVMe metrics from the EBS CSI driver.

Description of changes

New suite of tests that check that all metrics are emitted correctly with the correct dimension sets
New test environment using Terraform
I. Installs the agent using the Helm charts
ii. Installs the EBS CSI add-on and enables the metrics
iii. An Amazon Linux container is deployed with an ephemeral EBS volume attached to test the metrics
These tests do assert on the Type label (as do the other tests) which needs to be updated in Add translation logic for NVME metrics amazon-cloudwatch-agent#1625 (comment) to work.

Update 1:

Pinned the Helm version since it seems that versions > 3 have some breaking changes for us. This includes pod identity, entity, and of course, EBS.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

https://github.com/aws/amazon-cloudwatch-agent/actions/runs/14500675779/job/40745198026
I . The test ran with a dev build that has the EBS CSI Prometheus scraping logic
Latest test with a real build: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/15978524238/job/45067569331

zhihonl · 2025-04-23T19:41:38Z

+}
+
+func (t *DiskIOTestRunner) GetAgentRunDuration() time.Duration {
+	return 5 * time.Minute


Is 5 minutes necessary for NVME metrics? Can it be shorter or 5 is the most consistent we found

I found this to be the most consistent. I would've liked it to be shorter :(

lisguo · 2025-04-23T20:11:48Z

@@ -0,0 +1,62 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",


can you provide a sample emf log to cross reference?

{ "AutoScalingGroupName": "eks-dscia-ng-c8ca3ae1-cabd-aab3-a6d0-50ecc2aa8c30", "CloudWatchMetrics": [{ "Namespace": "ContainerInsights", "Dimensions": [ ["ClusterName"], ["ClusterName", "InstanceId", "NodeName"], ["ClusterName", "InstanceId", "NodeName", "VolumeId"] ], "Metrics": [{ "Name": "node_diskio_ebs_volume_performance_exceeded_tp", "Unit": "Second", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_total_read_bytes", "Unit": "Bytes", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_ec2_instance_performance_exceeded_tp", "Unit": "Second", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_volume_performance_exceeded_iops", "Unit": "Second", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_total_read_ops", "Unit": "Count", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_total_write_bytes", "Unit": "Bytes", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_total_write_ops", "Unit": "Count", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_total_write_time", "Unit": "Second", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_ec2_instance_performance_exceeded_iops", "Unit": "Second", "StorageResolution": 60 }, { "Name": "node_diskio_ebs_total_read_time", "Unit": "Second", "StorageResolution": 60 }] }], "ClusterName": "dscia-testing", "InstanceId": "i-0434a029434c980f7", "InstanceType": "m5.large", "NodeName": "ip-192-168-8-144.ec2.internal", "Timestamp": "1745510582682", "Type": "NodeNVME", "Version": "0", "VolumeId": "vol-0eb35c5cbb1ebfc97", "http.scheme": "http", "instance_id": "i-0434a029434c980f7", "k8s.namespace.name": "kube-system", "kubernetes": { "host": "ip-192-168-8-144.ec2.internal" }, "net.host.name": "ebs-csi-node.kube-system.svc", "net.host.port": "3302", "server.address": "ebs-csi-node.kube-system.svc", "server.port": "3302", "service.instance.id": "ebs-csi-node.kube-system.svc:3302", "service.name": "containerInsightsNVMeExporterScraper", "url.scheme": "http", "volume_id": "vol-0eb35c5cbb1ebfc97", "node_diskio_ebs_ec2_instance_performance_exceeded_iops": 0, "node_diskio_ebs_ec2_instance_performance_exceeded_tp": 0, "node_diskio_ebs_total_read_bytes": 1056768, "node_diskio_ebs_total_read_ops": 39, "node_diskio_ebs_total_read_time": 0.02430000000003929, "node_diskio_ebs_total_write_bytes": 135168, "node_diskio_ebs_total_write_ops": 23, "node_diskio_ebs_total_write_time": 0.020897999999988315, "node_diskio_ebs_volume_performance_exceeded_iops": 0, "node_diskio_ebs_volume_performance_exceeded_tp": 0 }

- Remove unnecessary operator image update - Add missing metric from schema

- Remove empty logfile which should make it easier to debug later down the line - Remove run_as_user since root is already default - Remove debug line

lisguo · 2025-07-01T22:48:28Z

+  }
+}
+
+resource "helm_release" "aws_observability" {


why is this called "helm_release"?

https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release

It’s just how it’s named for the Helm charts

lisguo · 2025-07-01T22:49:11Z

+        container {
+          name    = "app"
+          image   = "public.ecr.aws/amazonlinux/amazonlinux"
+          command = ["/bin/bash", "-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]


why is this necessary? To force a write on disk to produce a metric?

lisguo · 2025-07-01T22:49:54Z

+
+variable "ami_type" {
+  type    = string
+  default = "AL2_x86_64"


not related to your PR but at some point we should start testing with AL2023 since EKS recommends AL2023

https://docs.aws.amazon.com/eks/latest/userguide/al2023.html

duhminick and others added 3 commits April 17, 2025 14:38

Add tests for EBS CSI NVMe on EKS

12cdcfd

Merge branch 'main' into dominic-ebs-eks-pr

20d1bc5

Fix format

b690227

duhminick changed the title ~~Add tests for EBS CSI NVMe on EKS~~ [DO NOT MERGE] Add tests for EBS CSI NVMe on EKS Apr 21, 2025

duhminick marked this pull request as ready for review April 21, 2025 19:58

duhminick requested a review from a team as a code owner April 21, 2025 19:58

zhihonl reviewed Apr 23, 2025

View reviewed changes

lisguo reviewed Apr 23, 2025

View reviewed changes

duhminick and others added 3 commits April 23, 2025 16:57

Address comments

8f3cfb8

- Remove unnecessary operator image update - Add missing metric from schema

Merge branch 'main' into dominic-ebs-eks-pr

d4b49aa

Improve test config JSON

9663017

- Remove empty logfile which should make it easier to debug later down the line - Remove run_as_user since root is already default - Remove debug line

varunch77 previously approved these changes Apr 28, 2025

View reviewed changes

duhminick changed the title ~~[DO NOT MERGE] Add tests for EBS CSI NVMe on EKS~~ Add tests for EBS CSI NVMe on EKS Jun 2, 2025

lisguo suggested changes Jun 10, 2025

View reviewed changes

Comment thread test/metric_value_benchmark/eks_resources/util.go Outdated

duhminick and others added 2 commits June 30, 2025 11:48

Merge branch 'main' into dominic-ebs-eks-pr

1e67d09

Rename from NodeNVME to NodeEBS

e2a7068

duhminick dismissed varunch77’s stale review via e2a7068 June 30, 2025 16:05

duhminick added 3 commits June 30, 2025 16:18

Add target to test case gen

0d2577e

Pin Helm version for EBS and Pod Identity

c723424

Pin Helm version for entity

6839a31

lisguo approved these changes Jul 1, 2025

View reviewed changes

varunch77 approved these changes Jul 2, 2025

View reviewed changes

duhminick merged commit acd1943 into main Jul 5, 2025
2 checks passed

duhminick deleted the dominic-ebs-eks-pr branch July 5, 2025 20:15

Navigation Menu

Uh oh!

Add tests for EBS CSI NVMe on EKS - #507

Add tests for EBS CSI NVMe on EKS#507
duhminick merged 11 commits into
mainfrom
dominic-ebs-eks-pr

duhminick commented Apr 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

zhihonl Apr 23, 2025

Uh oh!

duhminick Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lisguo Apr 23, 2025

Uh oh!

duhminick Apr 24, 2025

Uh oh!

Uh oh!

lisguo Jul 1, 2025

Uh oh!

duhminick Jul 5, 2025

Uh oh!

lisguo Jul 1, 2025

Uh oh!

duhminick Jul 5, 2025

Uh oh!

lisguo Jul 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1,62 @@
		{
		"$schema": "http://json-schema.org/draft-04/schema#",

Uh oh!

Conversation

duhminick commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

Update 1:

License

Tests

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

duhminick commented Apr 17, 2025 •

edited

Loading