using configmap to inject runtime and csi sideacr by BH4AWS · Pull Request #219 · openkruise/agents

BH4AWS · 2026-03-24T14:23:34Z

Ⅰ. Describe what this PR does

The purpose of this PR is to use ConfigMap as the data source to automatically inject CSI mount or agent runtime configurations into the sandbox.

// ShouldInjectCsiMount is the annotation key for inject csi mount plugin container.
// If set, the csi sidecar will be injected into the pod when the sandbox is created.
// The csi mount sidecar is used to mount the remote oss/nas storage to the sandbox container.
ShouldInjectCsiMount = "agents.kruise.io/should-inject-csi-mount-plugin"

// ShouldInjectAgentRuntime is the annotation key for inject agent runtime sidecar in init container.
// If set, the agent runtime sidecar will be injected into the pod when the sandbox is created.
// Some binary tools which are contained in the init agent runtime container. These are the basic tools for sandbox running.
ShouldInjectAgentRuntime = "agents.kruise.io/should-inject-agent-runtime"

TODO:
This is the first patch. In next, for the second path, i will modify the code in sandbox-manager.

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Ⅳ. Special notes for reviews

furykerry · 2026-03-24T16:20:17Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+// parseCSIMountConfig parses the CSI sidecar injection configuration from raw ConfigMap data.
+// It extracts the value associated with KEY_CSI_INJECTION_CONFIG key and unmarshals it into
+// a SidecarInjectConfig struct. If the key is not present in the configRaw map, it returns
+// an empty SidecarInjectConfig without error. The configuration includes the main container
+// settings, CSI sidecar containers list, and volumes to be injected.
+//
+// Parameters:
+//   - ctx: context.Context for the operation (currently unused but kept for future extensibility)
+//   - configRaw: map[string]string containing the raw ConfigMap data with potential CSI config
+//
+// Returns:
+//   - SidecarInjectConfig: The parsed configuration containing main container, sidecars, and volumes
+//   - error: An error if JSON unmarshaling fails, nil otherwise
+func parseCSIMountConfig(ctx context.Context, configRaw map[string]string) (SidecarInjectConfig, error) {
+	log := logf.FromContext(ctx)
+	csiMountConfig := SidecarInjectConfig{}
+
+	configValue, exists := configRaw[KEY_CSI_INJECTION_CONFIG]
+	if !exists || configValue == "" {
+		log.Info("csi container config not found or empty, using default configuration")
+		return csiMountConfig, nil
+	}
+
+	err := json.Unmarshal([]byte(configRaw[KEY_CSI_INJECTION_CONFIG]), &csiMountConfig)
+	if err != nil {
+		log.Error(err, "failed to json unmarshal csi mount config")
+		return csiMountConfig, err
+	}
+	return csiMountConfig, nil
+}
+
+// parseAgentRuntimeConfig parses the agent runtime injection configuration from raw ConfigMap data.
+// It extracts the value associated with KEY_RUNTIME_INJECTION_CONFIG key and unmarshals it into
+// a SidecarInjectConfig struct. If the key is not present in the configRaw map, it returns
+// an empty SidecarInjectConfig without error. The configuration includes the main container
+// settings with lifecycle hooks, init runtime containers list, and volumes to be injected.
+//
+// Parameters:
+//   - ctx: context.Context for the operation (currently unused but kept for future extensibility)
+//   - configRaw: map[string]string containing the raw ConfigMap data with potential runtime config
+//
+// Returns:
+//   - SidecarInjectConfig: The parsed configuration containing main container, sidecars (init containers), and volumes
+//   - error: An error if JSON unmarshaling fails, nil otherwise
+func parseAgentRuntimeConfig(ctx context.Context, configRaw map[string]string) (SidecarInjectConfig, error) {
+	log := logf.FromContext(ctx)
+	csiMountConfig := SidecarInjectConfig{}
+
+	configValue, exists := configRaw[KEY_RUNTIME_INJECTION_CONFIG]
+	if !exists || configValue == "" {
+		log.Info("agent runtime config not found or empty, using default configuration")
+		return csiMountConfig, nil
+	}
+
+	err := json.Unmarshal([]byte(configValue), &csiMountConfig)
+	if err != nil {
+		log.Error(err, "failed to json unmarshal agent runtime config")
+		return csiMountConfig, err
+	}
+	return csiMountConfig, nil
+}
+
+// setCSIMountContainer injects CSI mount configurations into the SandboxTemplate's pod spec.
+// It configures the main container (first container in the spec) with CSI sidecar settings,
+// appends additional CSI sidecar containers, and mounts shared volumes.
+// Volumes are only added if they don't already exist in the template.
+func setCSIMountContainer(ctx context.Context, obj *corev1.PodTemplateSpec, config SidecarInjectConfig) {
+	log := logf.FromContext(ctx)
+
+	// set main container, the first container is the main container
+	if len(obj.Spec.Containers) == 0 {
+		log.Info("no found the template containers")
+		return
+	}
+
+	mainContainer := &obj.Spec.Containers[0]
+	setMainContainerWhenInjectCSISidecar(mainContainer, config)
+
+	// set csi sidecars
+	for _, csiSidecar := range config.Sidecars {
+		obj.Spec.Containers = append(obj.Spec.Containers, csiSidecar)
+	}
+
+	// set share volume
+	if len(config.Volumes) > 0 {
+		if obj.Spec.Volumes == nil {
+			obj.Spec.Volumes = make([]corev1.Volume, 0, len(config.Volumes))
+		}
+		for _, vol := range config.Volumes {
+			if findVolumeByName(obj.Spec.Volumes, vol.Name) {
+				continue
+			}
+			obj.Spec.Volumes = append(obj.Spec.Volumes, vol)
+		}
+	}
+}
+
+// setMainContainerWhenInjectCSISidecar configures the main container with environment variables and volume mounts from the CSI sidecar configuration.
+// It appends environment variables and volume mounts to the main container, skipping any that already exist (matched by name) to avoid duplicates.


plz merge two parse function into one to avoid code duplication

parseCSIMountConfig and parseAgentRuntimeConfig ---> parseInjectConfig function

furykerry · 2026-03-24T16:21:28Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+	obj.Spec.Volumes = append(obj.Spec.Volumes, config.Volumes...)
+}
+
+func setMainContainerConfigWhenInjectRuntimeSidecar(mainContainer *corev1.Container, config SidecarInjectConfig) {


plz merge two set function to avoid code duplication

The function is implemented as a sub-function and is not exposed for external calls; thus, it remains in its current implementation solely to simplify the implementation of the main function.

furykerry · 2026-03-24T16:27:25Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+			mainContainer.VolumeMounts = make([]corev1.VolumeMount, 0, len(config.MainContainer.VolumeMounts))
+		}
+		for _, volMount := range config.MainContainer.VolumeMounts {
+			if findVolumeMountByName(mainContainer.VolumeMounts, volMount.Name) {


also compare the mount path?

we will modify the implementation function in the next patch.

codecov · 2026-03-25T07:38:08Z

Codecov Report

❌ Patch coverage is 92.06349% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.47%. Comparing base (4d5825c) to head (5e6ce43).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/controller/sandbox/core/util.go	55.00%	6 Missing and 3 partials ⚠️
...g/controller/sandbox/core/sidecar_config_inject.go	99.04%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #219      +/-   ##
==========================================
+ Coverage   61.80%   62.47%   +0.67%     
==========================================
  Files         104      106       +2     
  Lines        6694     6827     +133     
==========================================
+ Hits         4137     4265     +128     
- Misses       2290     2294       +4     
- Partials      267      268       +1

Flag	Coverage Δ
unittests	`62.47% <92.06%> (+0.67%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

furykerry · 2026-03-25T22:00:21Z

api/v1alpha1/sandbox_types.go

+	// ShouldInjectCsiMount is the annotation key for inject csi mount plugin container.
+	// If set, the csi sidecar will be injected into the pod when the sandbox is created.
+	// The csi mount sidecar is used to mount the remote oss/nas storage to the sandbox container.
+	ShouldInjectCsiMount = "agents.kruise.io/should-inject-csi-mount-plugin"


agents.kruise.io/should-inject-csi-mount-plugin -> agents.kruise.io/inject-csi-plugin

furykerry · 2026-03-25T22:00:34Z

api/v1alpha1/sandbox_types.go

+	// ShouldInjectAgentRuntime is the annotation key for inject agent runtime sidecar in init container.
+	// If set, the agent runtime sidecar will be injected into the pod when the sandbox is created.
+	// Some binary tools which are contained in the init agent runtime container. These are the basic tools for sandbox running.
+	ShouldInjectAgentRuntime = "agents.kruise.io/should-inject-agent-runtime"


agents.kruise.io/should-inject-agent-runtime -> agents.kruise.io/inject-agent-runtime

furykerry · 2026-03-25T22:03:59Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+	}, config)
+	if err != nil {
+		if errors.IsNotFound(err) {
+			logger.Info("no found the injection configuration, using default")


no found the injection configuration, using default -> injection configuration not found, skip injection

furykerry · 2026-03-25T22:05:14Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+//   - error: An error if JSON unmarshaling fails, nil otherwise or when config key is missing/empty
+func parseInjectConfig(ctx context.Context, configKey string, configRaw map[string]string) (SidecarInjectConfig, error) {
+	log := logf.FromContext(ctx)
+	csiMountConfig := SidecarInjectConfig{}


csiMountConfig -> sidecarConfig

furykerry · 2026-03-25T22:06:47Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+
+	err := json.Unmarshal([]byte(configRaw[configKey]), &csiMountConfig)
+	if err != nil {
+		log.Error(err, "failed to json unmarshal csi mount config")


failed to unmarshal sidecar config for
log.Errorf(err, "failed to unmarshal sidecar config for the %v", configKey)

furykerry · 2026-03-25T22:08:55Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+
+	// set main container, the first container is the main container
+	if len(obj.Spec.Containers) == 0 {
+		log.Info("no found the template containers")


log.Info("no container found in sidecar template")

furykerry · 2026-03-25T22:10:15Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+	}
+
+	mainContainer := &obj.Spec.Containers[0]
+	setMainContainerWhenInjectCSISidecar(mainContainer, config)


should csi sidecar should also be a native sidecar ?

furykerry · 2026-03-25T22:11:13Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+	obj.Spec.InitContainers = append(obj.Spec.InitContainers, config.Sidecars...)
+
+	if len(obj.Spec.Containers) == 0 {
+		log.Info("no found the template container, default main container is the first container")


log.Info("no container found in sidecar template for agent runtime")

furykerry · 2026-03-25T22:11:26Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+// setAgentRuntimeContainer injects agent runtime configurations into the SandboxTemplate's pod spec.
+// It appends agent runtime containers as init containers and configures the main container (first container) with runtime settings.
+// The init containers run before the main containers to prepare the runtime environment.
+func setAgentRuntimeContainer(ctx context.Context, obj *corev1.PodTemplateSpec, config SidecarInjectConfig) {


plz rename obj to template

furykerry · 2026-03-25T22:14:30Z

pkg/controller/sandbox/core/sidecar_config_inject.go

+
+	// using config to override the main container lifecycle post start
+	if config.MainContainer.Lifecycle != nil && config.MainContainer.Lifecycle.PostStart != nil {
+		mainContainer.Lifecycle.PostStart = config.MainContainer.Lifecycle.PostStart


what if main container already have a post starthook, override the poststarthook may cause agent runs in correctly. Instead of silently overside postarthook, we should at least log an error and abort the injection

Signed-off-by: jicheng.sk <jicheng.sk@alibaba-inc.com>

furykerry

/lgtm
/approve

kruise-bot · 2026-03-27T16:48:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: furykerry

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kruise-bot added the size/XXL label Mar 24, 2026

BH4AWS force-pushed the feat/auto_inject_csi_runtime_config branch from 3cf2c3c to 8cb8853 Compare March 24, 2026 14:27

furykerry reviewed Mar 24, 2026

View reviewed changes

BH4AWS force-pushed the feat/auto_inject_csi_runtime_config branch from 8cb8853 to 521d1d7 Compare March 25, 2026 07:33

BH4AWS force-pushed the feat/auto_inject_csi_runtime_config branch from 521d1d7 to 54481a5 Compare March 25, 2026 07:57

furykerry reviewed Mar 25, 2026

View reviewed changes

using configmap to inject runtime and csi sideacr

5e6ce43

Signed-off-by: jicheng.sk <jicheng.sk@alibaba-inc.com>

BH4AWS force-pushed the feat/auto_inject_csi_runtime_config branch from 54481a5 to 5e6ce43 Compare March 27, 2026 02:45

furykerry approved these changes Mar 27, 2026

View reviewed changes

kruise-bot assigned furykerry Mar 27, 2026

kruise-bot added the lgtm label Mar 27, 2026

furykerry merged commit 0566fe6 into openkruise:master Mar 27, 2026
12 of 13 checks passed

Conversation

BH4AWS commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Ⅳ. Special notes for reviews

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

furykerry left a comment

Choose a reason for hiding this comment

Uh oh!

kruise-bot commented Mar 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BH4AWS commented Mar 24, 2026 •

edited

Loading

codecov bot commented Mar 25, 2026 •

edited

Loading