backups-restic addons, backups are not sent to AWS

[clickersmudge]

Hi,
Are there any additional steps needed to get backups sent to AWS?

Steps before installation

export HCLOUD_TOKEN=MY_HCLOUD_TOKEN
export AWS_ACCESS_KEY_ID=MY_AWS_ACCESS_KEY_ID // kubeone-backups
export AWS_SECRET_ACCESS_KEY=MY_AWS_SECRET_ACCESS_KEY // kubeone-backups

We have configured " backups-restic"

addons:
  enable: true
  addons:
    - name: backups-restic
      params:
        resticPassword: "my-restic-password"
        s3Bucket: "my-bucket"
        awsDefaultRegion: "my-region"

Backups are not sent.

[xmudrii]

@clickersmudge Are you still facing an issue with this one?

[clickersmudge]

@xmudrii Yes.

[clickersmudge]

@xmudrii

kubectl get cronjobs -A
NAMESPACE            NAME             SCHEDULE     SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cluster-bzfnxlqw2z   etcd-defragger   @every 3h    False     0        <none>          103m
cluster-dx8kj2dbtw   etcd-defragger   @every 3h    False     0        <none>          144m
cluster-zn229h7sv6   etcd-defragger   @every 3h    False     0        <none>          142m
kube-system          etcd-s3-backup   @every 30m   False     0        18m             26h
telemetry-system     telemetry-job    0 0 * * *    False     0        12h             26h

kubectl describe cronjob -n kube-system etcd-s3-backup
Name:                          etcd-s3-backup
Namespace:                     kube-system
Labels:                        kubeone.io/addon=backups-restic
Annotations:                   <none>
Schedule:                      @every 30m
Concurrency Policy:            Forbid
Suspend:                       False
Successful Job History Limit:  0
Failed Job History Limit:      1
Starting Deadline Seconds:     <unset>
Selector:                      <unset>
Parallelism:                   <unset>
Completions:                   <unset>
Pod Template:
  Labels:  <none>
  Init Containers:
   snapshoter:
    Image:      gcr.io/etcd-development/etcd:v3.4.3
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      set -euf
      mkdir -p /backup/pki/kubernetes
      mkdir -p /backup/pki/etcd
      cp -a /etc/kubernetes/pki/etcd/ca.crt /backup/pki/etcd/
      cp -a /etc/kubernetes/pki/etcd/ca.key /backup/pki/etcd/
      cp -a /etc/kubernetes/pki/ca.crt /backup/pki/kubernetes
      cp -a /etc/kubernetes/pki/ca.key /backup/pki/kubernetes
      cp -a /etc/kubernetes/pki/front-proxy-ca.crt /backup/pki/kubernetes
      cp -a /etc/kubernetes/pki/front-proxy-ca.key /backup/pki/kubernetes
      cp -a /etc/kubernetes/pki/sa.key /backup/pki/kubernetes
      cp -a /etc/kubernetes/pki/sa.pub /backup/pki/kubernetes
      etcdctl snapshot save /backup/etcd-snapshot.db
    Environment:
      ETCDCTL_API:           3
      ETCDCTL_DIAL_TIMEOUT:  3s
      ETCDCTL_CACERT:        /etc/kubernetes/pki/etcd/ca.crt
      ETCDCTL_CERT:          /etc/kubernetes/pki/etcd/healthcheck-client.crt
      ETCDCTL_KEY:           /etc/kubernetes/pki/etcd/healthcheck-client.key
      ETCD_HOSTNAME:          (v1:spec.nodeName)
    Mounts:
      /backup from etcd-backup (rw)
      /etc/kubernetes/pki from host-pki (ro)
  Containers:
   uploader:
    Image:      docker.io/restic/restic:0.9.6
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      set -euf
      restic snapshots -q || restic init -q
      restic backup --tag=etcd --host=${ETCD_HOSTNAME} /backup
      restic forget --prune --keep-last 48
    Environment:
      ETCD_HOSTNAME:           (v1:spec.nodeName)
      RESTIC_REPOSITORY:      kubermatic-kubeone-backups
      RESTIC_PASSWORD:        <set to the key 'password' in secret 'restic-config'>  Optional: false
      AWS_DEFAULT_REGION:     eu-central-1
      AWS_ACCESS_KEY_ID:      <set to the key 'AWS_ACCESS_KEY_ID' in secret 'kubeone-backups-credentials'>      Optional: false
      AWS_SECRET_ACCESS_KEY:  <set to the key 'AWS_SECRET_ACCESS_KEY' in secret 'kubeone-backups-credentials'>  Optional: false
    Mounts:
      /backup from etcd-backup (rw)
  Volumes:
   etcd-backup:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   host-pki:
    Type:            HostPath (bare host directory volume)
    Path:            /etc/kubernetes/pki
    HostPathType:    
Last Schedule Time:  Fri, 02 Dec 2022 13:37:18 +0100
Active Jobs:         <none>
Events:
  Type    Reason            Age   From                Message
  ----    ------            ----  ----                -------
  Normal  SuccessfulCreate  43m   cronjob-controller  Created job etcd-s3-backup-27833047
  Normal  SawCompletedJob   42m   cronjob-controller  Saw completed job: etcd-s3-backup-27833047, status: Complete
  Normal  SuccessfulDelete  42m   cronjob-controller  Deleted job etcd-s3-backup-27833047
  Normal  MissingJob        42m   cronjob-controller  Active job went missing: etcd-s3-backup-27833047
  Normal  SuccessfulCreate  12m   cronjob-controller  Created job etcd-s3-backup-27833077
  Normal  SawCompletedJob   12m   cronjob-controller  Saw completed job: etcd-s3-backup-27833077, status: Complete
  Normal  SuccessfulDelete  12m   cronjob-controller  Deleted job etcd-s3-backup-27833077
  Normal  MissingJob        12m   cronjob-controller  Active job went missing: etcd-s3-backup-27833077

We changed Cron every minute and try to catch pods during that time.

kubectl edit cronjob -n kube-system          etcd-s3-backup
   ...
   schedule: '@every 1m'
   ...

kubectl logs -n kube-system dx8kj2dbtw-backup-default-backups-create-cj4jx8krzp-xhdt8
Defaulted container "store-container" out of: store-container, backup-creator (init)
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
upload: '/backup/snapshot.db' -> 's3://kubermatic-backup-s3/dx8kj2dbtw-default-backups-2022-12-02t13-10-31.db' (13799456 bytes in 1.0 seconds, 13.84 MB/s) [1 of 1]

kubectl logs -n kube-system etcd-s3-backup-27833124-v6b6n
Defaulted container "uploader" out of: uploader, snapshoter (init)
Fatal: unable to open config file: Stat: stat kubermatic-kubeone-backups/config: no such file or directory
Is there a repository at the following location?
kubermatic-kubeone-backups
created new cache in /root/.cache/restic

Files:           9 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 46.135 MiB

processed 9 files, 46.135 MiB in 0:00
snapshot f6ceb8b4 saved
Applying Policy: keep the last 48 snapshots snapshots
keep 1 snapshots:
ID        Time                 Host                     Tags        Reasons        Paths
------------------------------------------------------------------------------------------
f6ceb8b4  2022-12-02 13:25:10  kubeone-control-plane-1  etcd        last snapshot  /backup
------------------------------------------------------------------------------------------
1 snapshots

[clickersmudge]

Hey,
We're back on topic.
We tested two scenarios.

Scenario 1

We have created a User, Bucket and Policy with full AWS S3 permissions.

export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=YOUR_AWS_DEFAULT_REGION
export RESTIC_PASSWORD=YOUR_RESTIC_PASSWORD
restic -r s3:s3.amazonaws.com/YOUR_BUCKET init

In the kubeone.yaml file, we set the parameters.

addons:
    # name of the addon to be enabled/deployed (e.g. backups-restic)
    - name: "backups-restic"
      # delete triggers deletion of the deployed addon
      delete: false
      # params is a key-value map of values passed to the addons templating engine,
      # to be used in the addon's manifests. Values defined here override the values
      # defined in globalParams.
      params:
        resticPassword: "YOUR_RESTICPASSWORD"
        s3Bucket: "YOUR_S3BUCKET"
        awsDefaultRegion: "YOUR_AWSDEFAULTREGION"

In the credentials.yaml file, we set the parameters.

HCLOUD_TOKEN: "YOUR_HCLOUD_TOKEN"
AWS_ACCESS_KEY_ID: "YOUR_AWS_ACCESS_KEY_ID"
AWS_SECRET_ACCESS_KEY: "YOUR_AWS_SECRET_ACCESS_KEY"

We installed KubeOne.

kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json

Error:

kubectl logs -n kube-system etcd-s3-backup-27892557-4xf6n
Defaulted container "uploader" out of: uploader, snapshoter (init)
Fatal: unable to open config file: Stat: stat xyz-backups-restic/config: no such file or directory
Is there a repository at the following location?
xyz-backups-restic
created new cache in /root/.cache/restic

Files:           9 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 8.643 MiB

processed 9 files, 8.643 MiB in 0:00
snapshot 6a4b2ddd saved
Applying Policy: keep the last 48 snapshots snapshots
keep 1 snapshots:
ID        Time                 Host                     Tags        Reasons        Paths
------------------------------------------------------------------------------------------
6a4b2ddd  2023-01-12 19:57:59  kubeone-control-plane-1  etcd        last snapshot  /backup
------------------------------------------------------------------------------------------
1 snapshots

Scenario 2

In scenario two, we deleted the repository from AWS so that the script could create it itself, but that didn't do anything. The bug is the same.

[clickersmudge]

We noticed a strange thing.

backups-restic.yaml from "1.5.4"

piVersion: batch/v1beta1
kind: CronJob
metadata:
  name: etcd-s3-backup
  namespace: kube-system
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  schedule: '@every 30m'
  successfulJobsHistoryLimit: 0
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          dnsPolicy: ClusterFirstWithHostNet
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
          - key: node-role.kubernetes.io/control-plane
            effect: NoSchedule
            operator: Exists
          - key: node-role.kubernetes.io/master
            effect: NoSchedule
            operator: Exists
          restartPolicy: OnFailure
          volumes:
          - name: etcd-backup
            emptyDir: {}
          - name: host-pki
            hostPath:
              path: /etc/kubernetes/pki
          initContainers:
          - name: snapshoter
            image: {{ Registry "gcr.io" }}/etcd-development/etcd:v3.4.3
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - |-
              set -euf
              mkdir -p /backup/pki/kubernetes
              mkdir -p /backup/pki/etcd
              cp -a /etc/kubernetes/pki/etcd/ca.crt /backup/pki/etcd/
              cp -a /etc/kubernetes/pki/etcd/ca.key /backup/pki/etcd/
              cp -a /etc/kubernetes/pki/ca.crt /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/ca.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/front-proxy-ca.crt /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/front-proxy-ca.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/sa.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/sa.pub /backup/pki/kubernetes
              etcdctl snapshot save /backup/etcd-snapshot.db
            env:
            - name: ETCDCTL_API
              value: "3"
            - name: ETCDCTL_DIAL_TIMEOUT
              value: 3s
            - name: ETCDCTL_CACERT
              value: /etc/kubernetes/pki/etcd/ca.crt
            - name: ETCDCTL_CERT
              value: /etc/kubernetes/pki/etcd/healthcheck-client.crt
            - name: ETCDCTL_KEY
              value: /etc/kubernetes/pki/etcd/healthcheck-client.key
            - name: ETCD_HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            volumeMounts:
            - mountPath: /backup
              name: etcd-backup
            - mountPath: /etc/kubernetes/pki
              name: host-pki
              readOnly: true
          containers:
          - name: uploader
            image: {{ Registry "docker.io" }}/restic/restic:0.9.6
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - |-
              set -euf
              restic snapshots -q || restic init -q
              restic backup --tag=etcd --host=${ETCD_HOSTNAME} /backup
              restic forget --prune --keep-last 48
            env:
            - name: ETCD_HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: RESTIC_REPOSITORY
              value: "{{ required "Please provide s3Bucket" .Params.s3Bucket }}"
            - name: RESTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: restic-config
                  key: password
            - name: AWS_DEFAULT_REGION
              value: "{{ required "Please provide awsDefaultRegion" .Params.awsDefaultRegion }}"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: AWS_ACCESS_KEY_ID
                  name: kubeone-backups-credentials
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: AWS_SECRET_ACCESS_KEY
                  name: kubeone-backups-credentials
            volumeMounts:
            - mountPath: /backup
              name: etcd-backup

But on Kubernetes is different

apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"batch/v1beta1","kind":"CronJob","metadata":{"annotations":{},"labels":{"kubeone.io/addon":"backups-restic"},"name":"etcd-s3-backup","namespace":"kube-system"},"spec":{"concurrencyPolicy":"Forbid","failedJobsHistoryLimit":1,"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"command":["/bin/sh","-c","set -euf\nrestic snapshots -q || restic init -q\nrestic backup --tag=etcd --host=${ETCD_HOSTNAME} /backup\nrestic forget --prune --keep-last 48"],"env":[{"name":"ETCD_HOSTNAME","valueFrom":{"fieldRef":{"fieldPath":"spec.nodeName"}}},{"name":"RESTIC_REPOSITORY","value":"backups-restic"},{"name":"RESTIC_PASSWORD","valueFrom":{"secretKeyRef":{"key":"password","name":"restic-config"}}},{"name":"AWS_DEFAULT_REGION","value":"eu-central-1"},{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"kubeone-backups-credentials"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"kubeone-backups-credentials"}}}],"image":"docker.io/restic/restic:0.9.6","imagePullPolicy":"IfNotPresent","name":"uploader","volumeMounts":[{"mountPath":"/backup","name":"etcd-backup"}]}],"dnsPolicy":"ClusterFirstWithHostNet","hostNetwork":true,"initContainers":[{"command":["/bin/sh","-c","set -euf\nmkdir -p /backup/pki/kubernetes\nmkdir -p /backup/pki/etcd\ncp -a /etc/kubernetes/pki/etcd/ca.crt /backup/pki/etcd/\ncp -a /etc/kubernetes/pki/etcd/ca.key /backup/pki/etcd/\ncp -a /etc/kubernetes/pki/ca.crt /backup/pki/kubernetes\ncp -a /etc/kubernetes/pki/ca.key /backup/pki/kubernetes\ncp -a /etc/kubernetes/pki/front-proxy-ca.crt /backup/pki/kubernetes\ncp -a /etc/kubernetes/pki/front-proxy-ca.key /backup/pki/kubernetes\ncp -a /etc/kubernetes/pki/sa.key /backup/pki/kubernetes\ncp -a /etc/kubernetes/pki/sa.pub /backup/pki/kubernetes\netcdctl snapshot save /backup/etcd-snapshot.db"],"env":[{"name":"ETCDCTL_API","value":"3"},{"name":"ETCDCTL_DIAL_TIMEOUT","value":"3s"},{"name":"ETCDCTL_CACERT","value":"/etc/kubernetes/pki/etcd/ca.crt"},{"name":"ETCDCTL_CERT","value":"/etc/kubernetes/pki/etcd/healthcheck-client.crt"},{"name":"ETCDCTL_KEY","value":"/etc/kubernetes/pki/etcd/healthcheck-client.key"},{"name":"ETCD_HOSTNAME","valueFrom":{"fieldRef":{"fieldPath":"spec.nodeName"}}}],"image":"gcr.io/etcd-development/etcd:v3.4.3","imagePullPolicy":"IfNotPresent","name":"snapshoter","volumeMounts":[{"mountPath":"/backup","name":"etcd-backup"},{"mountPath":"/etc/kubernetes/pki","name":"host-pki","readOnly":true}]}],"nodeSelector":{"node-role.kubernetes.io/control-plane":""},"restartPolicy":"OnFailure","tolerations":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Exists"}],"volumes":[{"emptyDir":{},"name":"etcd-backup"},{"hostPath":{"path":"/etc/kubernetes/pki"},"name":"host-pki"}]}}}},"schedule":"@every 30m","successfulJobsHistoryLimit":0,"suspend":false}}
  creationTimestamp: "2023-01-12T21:38:02Z"
  generation: 1
  labels:
    kubeone.io/addon: backups-restic
  name: etcd-s3-backup
  namespace: kube-system
  resourceVersion: "305993"
  uid: 739e8435-105e-4140-827e-98652edbb811
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
          - command:
            - /bin/sh
            - -c
            - |-
              set -euf
              restic snapshots -q || restic init -q
              restic backup --tag=etcd --host=${ETCD_HOSTNAME} /backup
              restic forget --prune --keep-last 48
            env:
            - name: ETCD_HOSTNAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: RESTIC_REPOSITORY
              value: backups-restic
            - name: RESTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: restic-config
            - name: AWS_DEFAULT_REGION
              value: eu-central-1
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: AWS_ACCESS_KEY_ID
                  name: kubeone-backups-credentials
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: AWS_SECRET_ACCESS_KEY
                  name: kubeone-backups-credentials
            image: docker.io/restic/restic:0.9.6
            imagePullPolicy: IfNotPresent
            name: uploader
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /backup
              name: etcd-backup
          dnsPolicy: ClusterFirstWithHostNet
          hostNetwork: true
          initContainers:
          - command:
            - /bin/sh
            - -c
            - |-
              set -euf
              mkdir -p /backup/pki/kubernetes
              mkdir -p /backup/pki/etcd
              cp -a /etc/kubernetes/pki/etcd/ca.crt /backup/pki/etcd/
              cp -a /etc/kubernetes/pki/etcd/ca.key /backup/pki/etcd/
              cp -a /etc/kubernetes/pki/ca.crt /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/ca.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/front-proxy-ca.crt /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/front-proxy-ca.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/sa.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/sa.pub /backup/pki/kubernetes
              etcdctl snapshot save /backup/etcd-snapshot.db
            env:
            - name: ETCDCTL_API
              value: "3"
            - name: ETCDCTL_DIAL_TIMEOUT
              value: 3s
            - name: ETCDCTL_CACERT
              value: /etc/kubernetes/pki/etcd/ca.crt
            - name: ETCDCTL_CERT
              value: /etc/kubernetes/pki/etcd/healthcheck-client.crt
            - name: ETCDCTL_KEY
              value: /etc/kubernetes/pki/etcd/healthcheck-client.key
            - name: ETCD_HOSTNAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            image: gcr.io/etcd-development/etcd:v3.4.3
            imagePullPolicy: IfNotPresent
            name: snapshoter
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /backup
              name: etcd-backup
            - mountPath: /etc/kubernetes/pki
              name: host-pki
              readOnly: true
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
          tolerations:
          - effect: NoSchedule
            key: node-role.kubernetes.io/control-plane
            operator: Exists
          - effect: NoSchedule
            key: node-role.kubernetes.io/master
            operator: Exists
          volumes:
          - emptyDir: {}
            name: etcd-backup
          - hostPath:
              path: /etc/kubernetes/pki
              type: ""
            name: host-pki
  schedule: '@every 30m'
  successfulJobsHistoryLimit: 0
  suspend: false
status: {}

It looks like an old version is installed there.

Our binary version of KubeOne.

{
  "kubeone": {
    "major": "1",
    "minor": "5",
    "gitVersion": "1.5.4",
    "gitCommit": "d814bd9007381d0eb06fefb4f7345a128b8f8666",
    "gitTreeState": "",
    "buildDate": "2022-12-12T09:37:23Z",
    "goVersion": "go1.19.4",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "54",
    "gitVersion": "v1.54.2",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}

[ahmedwaleedmalik]

Hi @clickersmudge, I just tested this addon and it's working fine for me. Are you still facing this issue?

I configured my cluster like this:

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: v1.25.4
cloudProvider:
  aws: {}
addons:
  enable: true
  addons:
  - name: backups-restic
    params:
      resticPassword: "my-restic-password"
      s3Bucket: "s3:https://s3.amazonaws.com/waleed-kubeone-backups"
      awsDefaultRegion: "eu-central-1"

You can ignore the API change from apiVersion: batch/v1beta1 to apiVersion: batch/v1 since that shouldn't affect anything.

[clickersmudge]

@ahmedwaleedmalik Thank you for your help.
The key is Protocol with endpoint.

addons.addons.name: "backups-restic".params.s3Bucket: "YOUR_S3BUCKET" # s3:YOUR_ENDPOINT/YOUR_BUCKET