[no-relnote] Add E2E for containerd #1313

ArangoGutierrez · 2025-09-23T17:54:37Z

No description provided.

Copilot

Pull Request Overview

This PR adds comprehensive E2E testing for containerd runtime configuration alongside existing Docker testing infrastructure. The changes introduce a nested container testing framework that allows running tests inside containers to validate NVIDIA Container Toolkit behavior in containerized environments.

Adds new E2E tests for containerd drop-in configuration functionality
Introduces nvidia-cdi-refresh systemd unit testing
Implements nested container runner infrastructure for isolated testing

Reviewed Changes

Copilot reviewed 9 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/go.mod	Adds new dependencies for UUID generation and test utilities
tests/e2e/runner.go	Implements nested container runner with Docker installation and CTK setup
tests/e2e/nvidia-ctk_containerd_test.go	New comprehensive containerd E2E test suite
tests/e2e/nvidia-ctk_docker_test.go	Refactors to use shared runner infrastructure and fixes macOS compatibility
tests/e2e/nvidia-cdi-refresh_test.go	New systemd unit tests for CDI refresh functionality
tests/e2e/nvidia-container-cli_test.go	Refactors to use nested container runner
tests/e2e/installer.go	Adds containerd installation template and additional flags support
tests/e2e/e2e_test.go	Centralizes test runner initialization in BeforeSuite
tests/e2e/Makefile	Documents new test categories

tests/e2e/nvidia-cdi-refresh_test.go

ArangoGutierrez · 2025-09-23T17:55:16Z

Builds on #1235

Doesn't include #1311 tests for that should be added as a follow up

coveralls · 2025-09-23T17:56:59Z

Pull Request Test Coverage Report for Build 18005738357

Details

0 of 1 (0.0%) changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.006%) to 36.277%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/config/engine/containerd/config_drop_in.go	0	1	0.0%

Totals
Change from base Build 17981864462:	0.006%
Covered Lines:	4827
Relevant Lines:	13306

💛 - Coveralls

ArangoGutierrez · 2025-09-24T10:44:36Z

I'll mark this PR as ready for review once #1235 is merged

ArangoGutierrez · 2025-09-25T07:13:46Z

I'll mark this PR as ready for review once #1235 is merged

Rebased

ArangoGutierrez · 2025-10-13T14:01:19Z

Rebased

elezar · 2025-10-13T14:41:46Z

tests/e2e/nvidia-ctk_containerd_test.go

+	AfterAll(func(ctx context.Context) {
+		// Cleanup: remove the container and the temporary script on the host.
+		// Use || true to ensure cleanup doesn't fail the test
+		runner.Run(fmt.Sprintf("docker rm -f %s 2>/dev/null || true", containerName)) //nolint:errcheck


Instead of the nolint let's just drop the return values.

Suggested change

runner.Run(fmt.Sprintf("docker rm -f %s 2>/dev/null || true", containerName)) //nolint:errcheck

_, _, _ = runner.Run(fmt.Sprintf("docker rm -f %s 2>/dev/null || true", containerName))

Does it mak sense to at least WARN if the cleanup fails? The || true doesn't ensure that the test doesn't fail, the fact that we don't check the return value does that.

elezar · 2025-10-13T14:43:21Z

tests/e2e/nvidia-ctk_containerd_test.go

+			 # Remove any imports line from the config (reset to original state)
+			 if [ -f /etc/containerd/config.toml ]; then
+				 grep -v "^imports = " /etc/containerd/config.toml > /tmp/config.toml.tmp && mv /tmp/config.toml.tmp /etc/containerd/config.toml || true
+			 fi


Why not just make a copy of the original config and restore that after / before each test?

elezar · 2025-10-13T14:43:37Z

tests/e2e/nvidia-ctk_containerd_test.go

+ 
+			 # Restart containerd to pick up the clean config
+			 systemctl restart containerd
+			 sleep 2


Is there a way to check containerd health?

elezar · 2025-10-13T14:45:23Z

tests/e2e/nvidia-ctk_containerd_test.go

+			output, _, err := nestedContainerRunner.Run(`cat /etc/containerd/conf.d/99-nvidia.toml`)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(output).To(ContainSubstring(`nvidia`))
+			Expect(output).To(ContainSubstring(`nvidia-cdi`))
+			Expect(output).To(ContainSubstring(`nvidia-legacy`))


As I mentioned in person, we are nolonger triggering the configuration of containerd with the current installation mechanism.

elezar · 2025-10-13T14:46:18Z

tests/e2e/nvidia-ctk_containerd_test.go

+			output, _, err = nestedContainerRunner.Run(`containerd config dump`)
+			Expect(err).ToNot(HaveOccurred())
+			// Verify imports section is in the merged config
+			Expect(output).To(ContainSubstring(`imports = ['/etc/containerd/conf.d/*.toml']`))


I actually think that config dump prints that ACTUAL paths of all files processed.

elezar · 2025-10-13T14:46:39Z

tests/e2e/nvidia-ctk_containerd_test.go

+				ContainSubstring(`default_runtime_name = "nvidia"`),
+				ContainSubstring(`default_runtime_name = 'nvidia'`),


These should definitely be VERSION specific checks.

Having tests that pass in multiple states are prone to be flaky. Please pick the format for the version of containerd that we have installed by default and use that. This makes the differences between SPECIFIC containerd versions more obvious when reading the tests.

elezar · 2025-10-13T14:47:31Z

tests/e2e/nvidia-ctk_containerd_test.go

+				ContainSubstring(`default_runtime_name = "nvidia"`),
+				ContainSubstring(`default_runtime_name = 'nvidia'`),
+			))
+			Expect(output).To(ContainSubstring(`enable_cdi = true`))


Where do we toggle this behaviour? It is disabled by default.

elezar · 2025-10-13T14:47:45Z

tests/e2e/nvidia-ctk_containerd_test.go

+				ContainSubstring(`[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]`),
+				ContainSubstring(`[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia]`),


Once again, thsi should be version-specific.

elezar · 2025-10-13T14:50:19Z

tests/e2e/nvidia-ctk_containerd_test.go

+	})
+
+	When("containerd already has a custom default runtime configured", func() {
+		It("should preserve the existing default runtime when --set-as-default=false is specified", func(ctx context.Context) {


--set-as-default=false is not specified. It is the default.

Note that we don't strictly speaking need a custom runtime to test the behaviour of NOT overriding the default.

elezar · 2025-10-13T14:50:52Z

tests/e2e/nvidia-ctk_containerd_test.go

+			 `)
+			Expect(err).ToNot(HaveOccurred())
+
+			// Configure containerd with drop-in config (explicitly not setting as default)


How are we configuring with the drop-in config in this case?

elezar · 2025-10-22T12:04:49Z

tests/e2e/nvidia-ctk_containerd_test.go

+					// Verify imports
+					if env.configVersion == 2 {
+						// containerd 1.7 shows actual resolved imports
+						err = validateImports(config, []string{"/etc/containerd/conf.d/99-nvidia.toml"}, true)


This is a sideeffect of the config dump command and does not reflect the actual content of the config file. One question I do have is why we're using validateImports here and not for the non v2 case?

elezar · 2025-10-22T12:05:26Z

tests/e2e/nvidia-ctk_containerd_test.go

+					Expect(err).ToNot(HaveOccurred(), "Failed to get plugin config")
+
+					// Verify CDI is enabled
+					cdiEnabled, err := getCDIEnabled(pluginConfig, env.configVersion)


Why is the version relevant here?

elezar · 2025-10-22T12:06:54Z

tests/e2e/nvidia-ctk_containerd_test.go

+	containerdSection := pluginConfig.Get("containerd")
+	if containerdSection == nil {
+		return "", fmt.Errorf("containerd section not found")
+	}
+
+	containerdTree, ok := containerdSection.(*toml.Tree)
+	if !ok {
+		return "", fmt.Errorf("containerd section is not a TOML tree")
+	}
+
+	defaultRuntime := containerdTree.Get("default_runtime_name")
+	if defaultRuntime == nil {
+		return "", nil // No default runtime set
+	}


Why don't we use GetPath? or Get("containerd.default_runtime_name")?

elezar · 2025-10-22T12:08:25Z

tests/e2e/nvidia-ctk_containerd_test.go

+}
+
+// validateRuntimeConfig validates a specific runtime configuration
+func validateRuntimeConfig(runtime interface{}, expectedType string, expectedOptions map[string]interface{}) error {


Under which conditions do we ever call this with anything other than a toml.Tree?

elezar · 2025-10-22T12:11:19Z

tests/e2e/nvidia-ctk_containerd_test.go

+	}
+
+	// Check runtime type
+	runtimeType, ok := runtimeMap["runtime_type"].(string)


This seems like something that we can check quite simply at the call site -- especially since we seem to mostly call this with runtimeType == "".

elezar · 2025-10-22T12:12:10Z

tests/e2e/nvidia-ctk_containerd_test.go

+		// For empty expectedType, we don't check the runtime type
+		// For nvidia runtime, we accept various runtime types
+		if expectedType == "nvidia" {
+			validTypes := []string{"io.containerd.runc.v2", "io.containerd.nvidia.v1", "io.containerd.nvidia.v2"}


When is this ever io.containerd.nvidia.v1? Where does this list come from?

elezar · 2025-10-22T12:12:29Z

tests/e2e/nvidia-ctk_containerd_test.go

+	if expectedType != "" {
+		// For empty expectedType, we don't check the runtime type
+		// For nvidia runtime, we accept various runtime types
+		if expectedType == "nvidia" {


We never seem to call this with expectedType == "nvidia" what is the purpose of this branch?

elezar · 2025-10-22T12:13:09Z

tests/e2e/nvidia-ctk_containerd_test.go

+			return fmt.Errorf("options is not a map or toml.Tree, got %T", v)
+		}
+
+		for key, expectedValue := range expectedOptions {


Could we use a gomega ContainsElements or ConsistsOf instead?

elezar · 2025-10-23T12:33:11Z

tests/e2e/nvidia-ctk_containerd_test.go

+				if !env.hasDefaultImports {
+					// For containerd 1.7.x, ensure a config file exists
+					// nvidia-ctk should add the imports directive automatically
+					_, _, err = nestedContainerRunner.Run(`
+if [ ! -f /etc/containerd/config.toml ]; then
+	mkdir -p /etc/containerd
+	containerd config default > /etc/containerd/config.toml
+fi`)
+					Expect(err).ToNot(HaveOccurred(), "Failed to create default containerd config")
+				}


Could you please explain why this is even required? The nvidia-ctk will create this file if it does not exist. It really should not be needed to create this before the tests.

elezar · 2025-10-23T12:34:14Z

tests/e2e/nvidia-ctk_containerd_test.go

+				// Ensure containerd is running
+				_, _, err = nestedContainerRunner.Run(`
+					# Start containerd if not running
+					if ! systemctl is-active --quiet containerd; then
+						systemctl start containerd
+						sleep 2
+					fi
+				`)
+				Expect(err).ToNot(HaveOccurred(), "Failed to ensure containerd is running")


Why do we need this IN addition to restartContainerdAndWait? Is there a reason that we can't just call that here if required?

elezar · 2025-10-23T12:35:32Z

tests/e2e/nvidia-ctk_containerd_test.go

+			if [ -f /tmp/containerd-config.toml.backup ]; then
+				cp /tmp/containerd-config.toml.backup /etc/containerd/config.toml
+			fi


Should we REMOVE /etc/containerd/config.toml if /tmp/containerd-config.toml.backup does not exist?

elezar · 2025-11-03T09:08:14Z

tests/e2e/nvidia-ctk_containerd_test.go

+			When("configuring containerd", func() {
+				It("should add NVIDIA runtime using drop-in config without modifying the main config", func(ctx context.Context) {
+					// Configure containerd using nvidia-ctk
+					_, _, err := nestedContainerRunner.Run(`nvidia-ctk runtime configure --runtime=containerd --config=/etc/containerd/config.toml --drop-in-config=/etc/containerd/conf.d/99-nvidia.toml --set-as-default --cdi.enabled`)


Just as a note: Should we use the defaults here for the --config and drop-in-config flags? The fact that we're explicitly setting the paths here is one reason that we missed the issue reported in #1404 (see #1407).

ArangoGutierrez · 2025-11-05T15:31:51Z

tests/e2e/runner.go


 	var additionalContainerArguments []string

+	if requiresGPUs {


Ha! nice, thanks for this Evan.

Signed-off-by: Evan Lezar <[email protected]>

This change makes the logic w.r.t mounting the host toolkit into a nested runner more explicit and adds a flag to indicate whether the runner needs GPU access. Signed-off-by: Evan Lezar <[email protected]>

Signed-off-by: Carlos Eduardo Arango Gutierrez <[email protected]>

elezar · 2025-11-06T13:35:18Z

tests/e2e/nvidia-ctk_containerd_test.go

+			})
+
+			When("configuring containerd", func() {
+				It("should add NVIDIA runtime using drop-in config and validate merged configuration", func(ctx context.Context) {


the "and validate merged configuration" is what we're doing in the test, not what we expect.

elezar · 2025-11-06T13:36:07Z

tests/e2e/nvidia-ctk_containerd_test.go

+					_, _, err := nestedContainerRunner.Run(`nvidia-ctk runtime configure --runtime=containerd --set-as-default --cdi.enabled`)
+					Expect(err).ToNot(HaveOccurred(), "Failed to configure containerd")


I think this is what determines what comes in the "It" statement.

elezar · 2025-11-06T13:36:56Z

tests/e2e/nvidia-ctk_containerd_test.go

+					Expect(mainConfigContents).To(ContainSubstring("imports"))
+					Expect(mainConfigContents).To(ContainSubstring("/etc/containerd/conf.d"))


If we want to check this, we should parse the toml.

elezar · 2025-11-06T13:39:16Z

tests/e2e/nvidia-ctk_containerd_test.go

+					imports := config.Get("imports")
+					Expect(imports).ToNot(BeNil())
+					importsList, ok := imports.([]interface{})
+					Expect(ok).To(BeTrue())
+					Expect(len(importsList)).To(BeNumerically(">=", 1))


Checking for the presence of imports is not sufficient. We need to ensure that we ALSO check that the desired folder (or file) is imported. For example, see the matcher here: https://github.com/ArangoGutierrez/nvidia-container-toolkit/pull/1/files#diff-056e23fcacb20f063e793331c10a8fedec7eedb12c88a3c7e857ad995ace49daR234-R249

(I'm not saying that we need to use a matcher, but that we should check the value properly).

ArangoGutierrez requested review from Copilot and elezar September 23, 2025 17:54

ArangoGutierrez self-assigned this Sep 23, 2025

Copilot AI reviewed Sep 23, 2025

View reviewed changes

tests/e2e/nvidia-cdi-refresh_test.go Outdated Show resolved Hide resolved

ArangoGutierrez requested a review from cdesiniotis September 23, 2025 17:56

ArangoGutierrez force-pushed the e2e_containerd branch from 5c85056 to 3dc480c Compare September 24, 2025 08:43

ArangoGutierrez force-pushed the e2e_containerd branch from 3dc480c to 563a192 Compare September 25, 2025 07:13

ArangoGutierrez force-pushed the e2e_containerd branch 3 times, most recently from 029af03 to 1899001 Compare September 25, 2025 11:16

ArangoGutierrez marked this pull request as ready for review September 25, 2025 11:26

elezar marked this pull request as draft October 13, 2025 12:10

ArangoGutierrez force-pushed the e2e_containerd branch 2 times, most recently from 51ad031 to c65e468 Compare October 13, 2025 13:57

ArangoGutierrez marked this pull request as ready for review October 13, 2025 14:01

elezar reviewed Oct 13, 2025

View reviewed changes

elezar reviewed Oct 22, 2025

View reviewed changes

ArangoGutierrez force-pushed the e2e_containerd branch from 34346af to 899fec9 Compare October 22, 2025 15:53

elezar reviewed Oct 23, 2025

View reviewed changes

ArangoGutierrez force-pushed the e2e_containerd branch from 899fec9 to f82172f Compare October 24, 2025 11:03

ArangoGutierrez requested a review from elezar October 27, 2025 08:07

elezar reviewed Nov 3, 2025

View reviewed changes

elezar force-pushed the e2e_containerd branch 3 times, most recently from 2498628 to 96059e8 Compare November 5, 2025 13:16

ArangoGutierrez commented Nov 5, 2025

View reviewed changes

tests/e2e/runner.go

var additionalContainerArguments []string

if requiresGPUs {

Copy link

Collaborator Author

ArangoGutierrez Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha! nice, thanks for this Evan.

ArangoGutierrez force-pushed the e2e_containerd branch from 96059e8 to 3a45cdf Compare November 6, 2025 11:24

ArangoGutierrez requested a review from elezar November 6, 2025 11:25

elezar and others added 3 commits November 6, 2025 14:21

[no-relnote] Support e2e tests on ARM64 systems

b041b64

Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Rework nested runner construction

31c2eb7

This change makes the logic w.r.t mounting the host toolkit into a nested runner more explicit and adds a flag to indicate whether the runner needs GPU access. Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Add E2E for containerd

241d330

Signed-off-by: Carlos Eduardo Arango Gutierrez <[email protected]>

elezar reviewed Nov 6, 2025

View reviewed changes

elezar force-pushed the e2e_containerd branch from 3a45cdf to 241d330 Compare November 6, 2025 13:35

elezar reviewed Nov 6, 2025

View reviewed changes

elezar mentioned this pull request Nov 6, 2025

Enable arm64 testing for contianer toolkit #1432

Open

	runner.Run(fmt.Sprintf("docker rm -f %s 2>/dev/null \|\| true", containerName)) //nolint:errcheck
	_, _, _ = runner.Run(fmt.Sprintf("docker rm -f %s 2>/dev/null \|\| true", containerName))

		ContainSubstring(`default_runtime_name = "nvidia"`),
		ContainSubstring(`default_runtime_name = 'nvidia'`),

		ContainSubstring(`[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]`),
		ContainSubstring(`[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia]`),

		_, _, err := nestedContainerRunner.Run(`nvidia-ctk runtime configure --runtime=containerd --set-as-default --cdi.enabled`)
		Expect(err).ToNot(HaveOccurred(), "Failed to configure containerd")

		Expect(mainConfigContents).To(ContainSubstring("imports"))
		Expect(mainConfigContents).To(ContainSubstring("/etc/containerd/conf.d"))

[no-relnote] Add E2E for containerd #1313

Are you sure you want to change the base?

[no-relnote] Add E2E for containerd #1313

Conversation

ArangoGutierrez commented Sep 23, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

ArangoGutierrez commented Sep 23, 2025

Uh oh!

coveralls commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18005738357

Details

💛 - Coveralls

Uh oh!

ArangoGutierrez commented Sep 24, 2025

Uh oh!

ArangoGutierrez commented Sep 25, 2025

Uh oh!

ArangoGutierrez commented Oct 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Sep 23, 2025 •

edited

Loading