Helm chart: Add support for encryption at rest configuration

[cmontemuino]

What would you like to be added:
The Helm chart should support configuring encryption at rest for Secrets in etcd, similar to how the binary installation and Karmada operator support it.

Why is this needed:

• Binary installation docs show encryption at rest configuration
• Karmada operator supports it via extraArgs/extraVolumes
• Helm chart is silent on this critical security feature
• Security-conscious users need etcd encryption without manual patching

[RainbowMango]

Thank you @cmontemuino for bringing this up.

It sounds reasonable. cc chart owners here to confirm. @seanlaii @zhzhuang-zju

[zhzhuang-zju]

@cmontemuino Thank you very much for your feedback. Is your request regarding support for configuring encryption at rest for Secrets in etcd similar to the following story?

karmada/docs/proposals/karmada-operator/api-server-sidecard-containers/README.md

Lines 54 to 124 in b978767

	#### Story 1
	As an infrastructure engineer, I want to integrate with my organization's KMS to configure encryption at rest for managed Karmada control planes.
	When using a KMS-based encryption provider, a KMS plugin is required to enable the integration. Once set up, the API server communicates to the plugin over a UNIX domain socket via gRPC.
	For that to work, when the API server is deployed as a pod as is the case when provisioned by the Karmada operator, the plugin has to run as a sidecar of the API server container.
	##### Sample Configuration
	```yaml
	---
	apiVersion: v1
	kind: ConfigMap
	metadata:
	name: encryption-config
	namespace: tenant1
	data:
	encryption-config.yaml: |
	apiVersion: apiserver.config.k8s.io/v1
	kind: EncryptionConfiguration
	resources:
	- resources:
	- secrets
	providers:
	- kms:
	apiVersion: v2
	name: custom-kms-provider
	endpoint: unix:///var/run/kmsplugin/socket.sock
	cachesize: 1000
	timeout: 3s
	- identity: {}
	---
	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: kube-apiserver
	namespace: tenant1
	labels:
	app: kube-apiserver
	spec:
	replicas: 1
	selector:
	matchLabels:
	app: kube-apiserver
	template:
	metadata:
	labels:
	app: kube-apiserver
	spec:
	containers:
	- name: kube-apiserver
	image: k8s.gcr.io/kube-apiserver:v1.21.0
	command:
	- kube-apiserver
	- --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
	volumeMounts:
	- name: encryption-config
	mountPath: /etc/kubernetes/encryption-config.yaml
	subPath: encryption-config.yaml
	- name: kms-socket
	mountPath: /var/run/kmsplugin
	- name: kms-plugin
	image: custom-kms-plugin:latest
	volumeMounts:
	- name: kms-socket
	mountPath: /var/run/kmsplugin
	volumes:
	- name: encryption-config
	configMap:
	name: encryption-config
	- name: kms-socket
	emptyDir: {}
	```

[zhzhuang-zju]

Just confirmed that Helm doesn't yet support the corresponding capability; it would be reasonable to add support for features similar to extraArgs/extraVolumes in the Karmada operator.

[cmontemuino]

@cmontemuino Thank you very much for your feedback. Is your request regarding support for configuring encryption at rest for Secrets in etcd similar to the following story?

karmada/docs/proposals/karmada-operator/api-server-sidecard-containers/README.md

Lines 54 to 124 in b978767

Story 1

As an infrastructure engineer, I want to integrate with my organization's KMS to configure encryption at rest for managed Karmada control planes.
When using a KMS-based encryption provider, a KMS plugin is required to enable the integration. Once set up, the API server communicates to the plugin over a UNIX domain socket via gRPC.
For that to work, when the API server is deployed as a pod as is the case when provisioned by the Karmada operator, the plugin has to run as a sidecar of the API server container.

Sample Configuration

--- 
apiVersion: v1 
kind: ConfigMap 
metadata: 
  name: encryption-config 
  namespace: tenant1 
data: 
  encryption-config.yaml: | 
    apiVersion: apiserver.config.k8s.io/v1 
    kind: EncryptionConfiguration 
    resources: 
      - resources: 
          - secrets 
        providers: 
          - kms: 
              apiVersion: v2 
              name: custom-kms-provider 
              endpoint: unix:///var/run/kmsplugin/socket.sock 
              cachesize: 1000 
              timeout: 3s 
          - identity: {} 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: kube-apiserver 
  namespace: tenant1 
  labels: 
    app: kube-apiserver 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: kube-apiserver 
  template: 
    metadata: 
      labels: 
        app: kube-apiserver 
    spec: 
      containers: 
        - name: kube-apiserver 
          image: k8s.gcr.io/kube-apiserver:v1.21.0 
          command: 
            - kube-apiserver 
            - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml 
          volumeMounts: 
            - name: encryption-config 
              mountPath: /etc/kubernetes/encryption-config.yaml 
              subPath: encryption-config.yaml 
            - name: kms-socket 
              mountPath: /var/run/kmsplugin 
        - name: kms-plugin 
          image: custom-kms-plugin:latest 
          volumeMounts: 
            - name: kms-socket 
              mountPath: /var/run/kmsplugin 
      volumes: 
        - name: encryption-config 
          configMap: 
            name: encryption-config 
        - name: kms-socket 
          emptyDir: {} 

Thanks for the quick response @zhzhuang-zju!

I don't think this is related to the sidecar containers proposal. If I understand it correctly, that proposal is about adding auxiliary containers alongside the API server (metrics exporters, log collectors, etc.), right?

My request is specifically about configuring the karmada-apiserver container itself with --encryption-provider-config flag.

The binary installation docs already show this configuration:
https://karmada.io/docs/installation/install-binary/#create-the-karmada-kubeconfig-files-and-etcd-encryption-key

And the Karmada operator supports it via extraArgs/extraVolumes (as you mentioned).

So this is more about exposing existing API server configuration options in the Helm chart, rather than adding new containers.

[cmontemuino]

I'd like to propose a solution that supports two modes to accommodate both PoC and production use cases:

Mode 1: Inline Secret Generation (PoC-friendly)

• Helm automatically generates encryption Secret during install
• Simple one-command setup
• ⚠️ Key visible in Helm values/history (not for production)

Mode 2: External Secret Reference (Production-ready)

• Users manage encryption Secret externally (Vault, Sealed Secrets, External Secrets Operator)
• Keys never stored in git or Helm values
• Supports key rotation

Changes to charts/karmada/values.yaml would look like this:

apiserver:
  encryption:
    enabled: false
    
    # Mode 1: Auto-generate Secret (testing/PoC)
    inline:
      enabled: false
      secretName: karmada-encryption-config
      configPath: /etc/karmada/encryption-config.yaml
      # Optional: provide pre-generated key
      # key: ""
    
    # Mode 2: Reference external Secret (production)
    existingSecret:
      enabled: false
      name: ""
      key: encryption-config.yaml

Would this approach work? I'm happy to open a PR if the direction looks good!

[zhzhuang-zju]

I don't think this is related to the sidecar containers proposal. If I understand it correctly, that proposal is about adding auxiliary containers alongside the API server (metrics exporters, log collectors, etc.), right?

Thanks for your clarification.

The "sidecar containers" proposal actually encompasses a suite of three capabilities:

• extraArgs: Used to configure the --encryption-provider-config flag for the karmada-apiserver.
• extraVolumes: Used to mount the encryption-at-rest configuration file.
• sidecars: Used to support KMS-based encryption by running the kms plugin as a sidecar container alongside the API server. (link: Add Support for API Server Sidecar Containers in Karmada Operator #6096)

My understanding is that for standard encryption-at-rest, having extraArgs and extraVolumes would be sufficient on their own. Is that correct?

Would this approach work? I'm happy to open a PR if the direction looks good!

Great! This is a very reasonable requirement. One thought: instead of a specific encryption feature, could we provide generic extraArgs and extraVolumes capabilities—similar to what karmada-operator offers—to make it more versatile? This way, encryption at rest would be just one of many possible use cases. WDYT?

[cmontemuino]

Great! This is a very reasonable requirement. One thought: instead of a specific encryption feature, could we provide generic extraArgs and extraVolumes capabilities—similar to what karmada-operator offers—to make it more versatile? This way, encryption at rest would be just one of many possible use cases. WDYT?

I'll explore it and come back with some insights.