[BUG] Unable to use AWS IAM auth for S3 storage in BackupRepo resource

[yogeshcbsi]

Describe the bug
I would like to use AWS IAM auth using IRSA to define BackupRepo but BackupRepo resource is not allowing to override env_auth configuration for AWS auth which allows to use AWS IAM authentication. Allow to overriding this configuration.

Most like because env_auth is hard-corded here

To Reproduce
Create a backup repo with env_auth = true. BackupRepo Pre-check fails even after using IAM role with sufficient permissions.

apiVersion: dataprotection.kubeblocks.io/v1alpha1
kind: BackupRepo
metadata:
  name: kubeblocks-backuprepo-test
  namespace: kb-system
spec:
  accessMethod: Tool
  config:
    bucket: kubeblocks-backup-repo
    endpoint: ''
    env_auth: 'true'
    region: us-east-1
  ....
  pvReclaimPolicy: Retain
  storageProviderRef: s3
  volumeCapacity: 100Gi

Expected behavior
env_auth: 'true' config value should be passed on rclone to allow AWS IAM instead of forcing to use self managed secret keys

[shanshanying]

@zjx20 could you pls help on this issue.

[zjx20]

@yogeshcbsi Thank you for reaching out.

As you may already know, this is a reminder that the ServiceAccount used by KubeBlocks DataProtection is named kubeblocks-dataprotection-worker, and it may exist in multiple namespaces (if you have multiple database clusters distributed across multiple namespaces).

Assuming you already know how to set up IRSA, you can create a StorageProvider CR yourself to use the env_auth option, for example:

apiVersion: dataprotection.kubeblocks.io/v1alpha1
kind: StorageProvider
metadata:
  name: s3-env-auth
spec:
  csiDriverName: ru.yandex.s3.csi
  csiDriverSecretTemplate: |
    accessKeyID: {{ index .Parameters "accessKeyId" }}
    secretAccessKey: {{ index .Parameters "secretAccessKey" }}
    {{- $region := index .Parameters "region" }}
    {{- $endpoint := index .Parameters "endpoint" }}
    {{- if not $endpoint }}
      {{- if hasPrefix "cn-" $region }}
        {{- $endpoint = (printf "https://s3.%s.amazonaws.com.cn" $region) }}
      {{- else }}
        {{- $endpoint = (printf "https://s3.%s.amazonaws.com" $region) }}
      {{- end }}
    {{- end }}
    endpoint: {{ $endpoint }}
  datasafedConfigTemplate: |
    [storage]
    type = s3
    provider = AWS
    env_auth = true
    access_key_id = {{ index .Parameters "accessKeyId" }}
    secret_access_key = {{ index .Parameters "secretAccessKey" }}
    region = {{ index .Parameters "region" }}
    endpoint = {{ index .Parameters "endpoint" }}
    root = {{ index .Parameters "bucket" }}
    no_check_certificate = {{ index .Parameters "insecure" }}
    no_check_bucket = {{ index .Parameters "noCheckBucket" }}
    chunk_size = 50Mi
  parametersSchema:
    credentialFields:
    - accessKeyId
    - secretAccessKey
    openAPIV3Schema:
      properties:
        accessKeyId:
          description: AWS access key
          type: string
        bucket:
          description: S3 bucket
          type: string
        endpoint:
          description: S3 endpoint (optional)
          type: string
        geesefsMemoryLimit:
          default: 512
          description: The value of --memory-limit parameter for geesefs, in MB
          type: integer
        geesefsReadAheadLarge:
          default: 20480
          description: The value of --read-ahead-large parameter for geesefs, in KB
          type: integer
        insecure:
          default: false
          description: Do not verify the server SSL certificate
          type: boolean
        mountOptions:
          description: Extra mount options for geesefs
          type: string
        noCheckBucket:
          default: false
          description: Do not check if the bucket exists, and do not try to create
            it
          type: boolean
        region:
          description: AWS region, e.g. us-west-1
          type: string
        secretAccessKey:
          description: AWS secret key
          type: string
      required:
      - bucket
      - region
      - accessKeyId
      - secretAccessKey
      type: object
  storageClassTemplate: |
    provisioner: ru.yandex.s3.csi
    parameters:
      mounter: geesefs
      # you can set mount options here, for example limit memory cache size (recommended)
      options: {{ printf "--memory-limit %s --read-ahead-large %s --dir-mode 0777 --file-mode 0666 %s --region %s" .Parameters.geesefsMemoryLimit .Parameters.geesefsReadAheadLarge .Parameters.mountOptions .Parameters.region }}
      bucket: {{ index .Parameters "bucket" }}
      csi.storage.k8s.io/provisioner-secret-name: {{ .CSIDriverSecretRef.Name }}
      csi.storage.k8s.io/provisioner-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}
      csi.storage.k8s.io/controller-publish-secret-name: {{ .CSIDriverSecretRef.Name }}
      csi.storage.k8s.io/controller-publish-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}
      csi.storage.k8s.io/node-stage-secret-name: {{ .CSIDriverSecretRef.Name }}
      csi.storage.k8s.io/node-stage-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}
      csi.storage.k8s.io/node-publish-secret-name: {{ .CSIDriverSecretRef.Name }}
      csi.storage.k8s.io/node-publish-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}

Then, you can create a BackupRepo CR using s3-env-auth. This should work theoretically, but I haven't actually tested it. If you encounter any issues, please feel free to provide further feedback.

[yogeshcbsi]

@yogeshcbsi Thank you for reaching out.

As you may already know, this is a reminder that the ServiceAccount used by KubeBlocks DataProtection is named kubeblocks-dataprotection-worker, and it may exist in multiple namespaces (if you have multiple database clusters distributed across multiple namespaces).

Assuming you already know how to set up IRSA, you can create a StorageProvider CR yourself to use the env_auth option, for example:

apiVersion: dataprotection.kubeblocks.io/v1alpha1
kind: StorageProvider
metadata:
name: s3-env-auth
spec:
csiDriverName: ru.yandex.s3.csi
csiDriverSecretTemplate: |
accessKeyID: {{ index .Parameters "accessKeyId" }}
secretAccessKey: {{ index .Parameters "secretAccessKey" }}
{{- $region := index .Parameters "region" }}
{{- $endpoint := index .Parameters "endpoint" }}
{{- if not $endpoint }}
{{- if hasPrefix "cn-" $region }}
{{- $endpoint = (printf "https://s3.%s.amazonaws.com.cn" $region) }}
{{- else }}
{{- $endpoint = (printf "https://s3.%s.amazonaws.com" $region) }}
{{- end }}
{{- end }}
endpoint: {{ $endpoint }}
datasafedConfigTemplate: |
[storage]
type = s3
provider = AWS
env_auth = true
access_key_id = {{ index .Parameters "accessKeyId" }}
secret_access_key = {{ index .Parameters "secretAccessKey" }}
region = {{ index .Parameters "region" }}
endpoint = {{ index .Parameters "endpoint" }}
root = {{ index .Parameters "bucket" }}
no_check_certificate = {{ index .Parameters "insecure" }}
no_check_bucket = {{ index .Parameters "noCheckBucket" }}
chunk_size = 50Mi
parametersSchema:
credentialFields:
- accessKeyId
- secretAccessKey
openAPIV3Schema:
properties:
accessKeyId:
description: AWS access key
type: string
bucket:
description: S3 bucket
type: string
endpoint:
description: S3 endpoint (optional)
type: string
geesefsMemoryLimit:
default: 512
description: The value of --memory-limit parameter for geesefs, in MB
type: integer
geesefsReadAheadLarge:
default: 20480
description: The value of --read-ahead-large parameter for geesefs, in KB
type: integer
insecure:
default: false
description: Do not verify the server SSL certificate
type: boolean
mountOptions:
description: Extra mount options for geesefs
type: string
noCheckBucket:
default: false
description: Do not check if the bucket exists, and do not try to create
it
type: boolean
region:
description: AWS region, e.g. us-west-1
type: string
secretAccessKey:
description: AWS secret key
type: string
required:
- bucket
- region
- accessKeyId
- secretAccessKey
type: object
storageClassTemplate: |
provisioner: ru.yandex.s3.csi
parameters:
mounter: geesefs
# you can set mount options here, for example limit memory cache size (recommended)
options: {{ printf "--memory-limit %s --read-ahead-large %s --dir-mode 0777 --file-mode 0666 %s --region %s" .Parameters.geesefsMemoryLimit .Parameters.geesefsReadAheadLarge .Parameters.mountOptions .Parameters.region }}
bucket: {{ index .Parameters "bucket" }}
csi.storage.k8s.io/provisioner-secret-name: {{ .CSIDriverSecretRef.Name }}
csi.storage.k8s.io/provisioner-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}
csi.storage.k8s.io/controller-publish-secret-name: {{ .CSIDriverSecretRef.Name }}
csi.storage.k8s.io/controller-publish-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}
csi.storage.k8s.io/node-stage-secret-name: {{ .CSIDriverSecretRef.Name }}
csi.storage.k8s.io/node-stage-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}
csi.storage.k8s.io/node-publish-secret-name: {{ .CSIDriverSecretRef.Name }}
csi.storage.k8s.io/node-publish-secret-namespace: {{ .CSIDriverSecretRef.Namespace }}
Then, you can create a BackupRepo CR using s3-env-auth. This should work theoretically, but I haven't actually tested it. If you encounter any issues, please feel free to provide further feedback.

I thought creating separate StorageProvider but wasn't sure if BackupRepo and/or operator can support it. Thanks for the suggestion @zjx20 ! I will give it try.