Skip to content

Added versioning configuration for COS buckets in CSI driver #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

Prachi03510
Copy link

Adds support for enabling versioning on COS buckets via the IBM COS CSI driver

if val, ok := secretMap["BucketVersioning"]; ok && val != "" {
enable := strings.ToLower(strings.TrimSpace(val))
if enable != "true" && enable != "false" {
return nil, status.Error(codes.InvalidArgument, fmt.Sprintf("Invalid BucketVersioning value in secret: %s. Must be 'true' or 'false'", val))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please log secret name as well her.
status.Error(codes.InvalidArgument, fmt.Sprintf("Invalid BucketVersioning value in secret: %s. Value set %s. Must be 'true' or 'false'", secretName, val))

fmt.Sprintf("Invalid BucketVersioning value in storage class: %s. Must be 'true' or 'false'", val))
}
BucketVersioning = enable
klog.Infof("BucketVersioning set via storage class: %s", BucketVersioning)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

klog.Infof("BucketVersioning value that will be set via storage class params: %s", BucketVersioning)

@@ -48,6 +48,7 @@ func (cs *controllerServer) CreateVolume(_ context.Context, req *csi.CreateVolum
kpRootKeyCrn string
pvcName string
pvcNamespace string
BucketVersioning string
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bucketVersioning

return nil, status.Error(codes.InvalidArgument, fmt.Sprintf("Invalid BucketVersioning value in secret: %s. Must be 'true' or 'false'", val))
}
BucketVersioning = enable
klog.Infof("BucketVersioning set via secret: %s", BucketVersioning)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

klog.Infof("BucketVersioning value that will be set via secret: %s", BucketVersioning)

@@ -441,6 +479,7 @@ func parseCustomSecret(secret *v1.Secret) map[string]string {
iamEndpoint string
cosEndpoint string
locationConstraint string
BucketVersioning string
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bucketVersioning

@@ -479,6 +518,10 @@ func parseCustomSecret(secret *v1.Secret) map[string]string {
locationConstraint = string(bytesVal)
}

if bytesVal, ok := secret.Data["BucketVersioning"]; ok {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bucketVersioning -> small case in the start to comply with how other params are set in the secret.

@Prachi03510
Copy link
Author

Prachi03510 commented May 20, 2025

Versioning enabled via secret for volume - s3fs

Positive Test Scenarios:

  1. Successfully created PVC with versioning enabled.
    cos-s3-csi-s3fs-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-secret3
  namespace: default
data:
  bucketName: cHJhY2hpLWRldnRlc3QwMS1idWNrZXQyMzk=
  accessKey: xxx
  secretKey: yyy
  versioningEnabled: dHJ1ZQ==
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-pvc-1
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-secret3"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml                                             
secret/cos-s3fs-secret3 created

ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml                                                
persistentvolumeclaim/cos-s3fs-pvc-1 created

ibm-object-csi-driver % kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-pvc-1   Bound    pvc-d474e0e3-e1c1-40f5-aeef-8df11b847e77  256Mi      RWO            ibm-object-storage-standard-s3fs   <unset>                 6s
image
  1. Successfully created PVC with versioning disabled.
    cos-s3-csi-s3fs-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-secret4
  namespace: default
data:
  bucketName: cHJhLXNlY3Jlbi1idWNrZXRuYW1l
  accessKey: xxx
  secretKey: yyy
  versioningEnabled: ZmFsc2U=
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-pvc-2
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-secret4"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml
secret/cos-s3fs-secret4 created

ibm-object-csi-driver % kubectl describe secret cos-s3fs-secret06
Name:         cos-s3fs-secret4
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  cos-s3-csi-driver

Data
====
mountOptions:       0 bytes
secretKey:          48 bytes
versioningEnabled:  5 bytes
accessKey:          32 bytes
bucketName:         18 bytes

ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml
persistentvolumeclaim/cos-s3fs-pvc-2 created

ibm-object-csi-driver % kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-pvc-2   Bound    pvc-c8348c5c-d5ae-42d9-adbb-653a06841120   256Mi      RWO            ibm-object-storage-standard-s3fs   <unset>                 8s
image
  1. Successfully updated PVC to enable versioning for an existing bucket.
    secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-secret5
  namespace: default
data:
  bucketName: cHJhLXNlY3Jlbi1idWNrZXRuYW1l
  accessKey: xxx
  secretKey: yyy
  BucketVersioning: dHJ1ZQ==
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-pvc-3
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-secret5"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
prachishivanandanure@Prachis-MacBook-Pro ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml
secret/cos-s3fs-secret5 created
prachishivanandanure@Prachis-MacBook-Pro ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml   
persistentvolumeclaim/cos-s3fs-pvc-3 created
prachishivanandanure@Prachis-MacBook-Pro ibm-object-csi-driver % kubectl get pvc
NAME                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-pvc-3      Bound      pvc-000fe610-a1db-4221-8bfa-14eb3cb0731e    256Mi      RWO            ibm-object-storage-standard-s3fs   <unset>                 2m33s
image

@Prachi03510
Copy link
Author

Negative Test Scenarios:

  1. Invalid Base64 Encoded Fields in Secret
    cos-s3-csi-s3fs-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-invalid-secret
  namespace: default
data:
  accessKey: !!invalidbase64==
  secretKey: yyy
  bucketName: cHJhLXNlY3Jlbi1idWNrZXRuYW1lbmFibGU=
 BucketVersioning: dHJ1ZQ==

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-invalid-pvc
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-invalid-secret"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml
secret/cos-s3fs-invalid-secret created

ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml
persistentvolumeclaim/cos-s3fs-invalid-pvc created

ibm-object-csi-driver % kubectl get pvc                                                  
NAME                   STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-invalid-pvc   Pending                                                                        ibm-object-storage-standard-s3fs   <unset>                 5s

ibm-object-csi-driver % kubectl describe pvc cos-s3fs-invalid-pvc 
% kubectl describe pvc cos-s3fs-invalid-pvc
Name:          cos-s3fs-invalid-pvc
Namespace:     default
StorageClass:  ibm-object-storage-standard-s3fs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   cos.csi.driver/secret: cos-s3fs-invalid-secret
               cos.csi.driver/secret-namespace: default
               volume.beta.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age                From                                                                                               Message
  ----     ------                ----               ----                                                                                               -------
  Normal   ExternalProvisioning  14s (x2 over 19s)  persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'cos.s3.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          4s (x5 over 19s)   cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  External provisioner is provisioning volume for claim "default/cos-s3fs-invalid-pvc"
  Warning  ProvisioningFailed    4s (x5 over 19s)   cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  failed to provision volume with StorageClass "ibm-object-storage-standard-s3fs": rpc error: code = InvalidArgument desc = Error in getting credentials rpc error: code = Unauthenticated desc = Valid access credentials are not provided in the secret| serviceId/accessKey/secretKey unknown
  1. Missing Required Field (no accessKey)
    cos-s3-csi-s3fs-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-miss-secret
  namespace: default
data:
  secretKey: ZDJiOTMzYWMzMTg4Mzc2YWRlMzE5OGZlMjZhZTIxYTI1ODJmNWMwYzg5MGJlNGVm
  bucketName: cHNyY29kZS1kZXZ0ZXN0
  BucketVersioning: ZmFsc2U=

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-miss-pvc
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-miss-secret"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml
secret/cos-s3fs-miss-secret created

% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml   
persistentvolumeclaim/cos-s3fs-miss-pvc created

% kubectl get pvc                                                  
NAME                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-miss-pvc   Pending                                                                        ibm-object-storage-standard-s3fs   <unset>                 21s

% kubectl describe pvc cos-s3fs-miss-pvc
Name:          cos-s3fs-miss-pvc
Namespace:     default
StorageClass:  ibm-object-storage-standard-s3fs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   cos.csi.driver/secret: cos-s3fs-miss-secret
               cos.csi.driver/secret-namespace: default
               volume.beta.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age               From                                                                                               Message
  ----     ------                ----              ----                                                                                               -------
  Normal   ExternalProvisioning  5s (x2 over 19s)  persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'cos.s3.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          4s (x5 over 19s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  External provisioner is provisioning volume for claim "default/cos-s3fs-miss-pvc"
  Warning  ProvisioningFailed    4s (x5 over 19s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  failed to provision volume with StorageClass "ibm-object-storage-standard-s3fs": rpc error: code = InvalidArgument desc = Error in getting credentials rpc error: code = Unauthenticated desc = Valid access credentials are not provided in the secret| serviceId/accessKey/secretKey unknown
  1. Invalid Bucket Name (uppercase not allowed)
    cos-s3-csi-s3fs-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-bucket-secret
  namespace: default
data:
  bucketName: U3RvcmFnZUJ1Y2tldFdpdGhVcHBlcmNhc2U=
  accessKey: xxx
  secretKey: yyy
  # BucketVersioning: ZmFsc2U=
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-bucket-pvc
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-bucket-secret"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml
secret/cos-s3fs-bucket-secret created

% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml   
persistentvolumeclaim/cos-s3fs-bucket-pvc created

% kubectl get pvc                                                  
NAME                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-bucket-pvc      Pending                                                                        ibm-object-storage-standard-s3fs   <unset>                 3s

14eb3cb0731e   256Mi      RWO            ibm-object-storage-standard-s3fs   <unset>                 40m
prachishivanandanure@Prachis-MacBook-Pro ibm-object-csi-driver % kubectl describe pvc cos-s3fs-bucketss-pvc                       
Name:          cos-s3fs-bucketss-pvc
Namespace:     default
StorageClass:  ibm-object-storage-standard-s3fs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   cos.csi.driver/secret: cos-s3fs-bucket-secret
               cos.csi.driver/secret-namespace: default
               volume.beta.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age               From                                                                                               Message
  ----     ------                ----              ----                                                                                               -------
  Normal   ExternalProvisioning  8s (x2 over 22s)  persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'cos.s3.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          5s (x5 over 22s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  External provisioner is provisioning volume for claim "default/cos-s3fs-bucketss-pvc"
  Warning  ProvisioningFailed    5s (x5 over 21s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  failed to provision volume with StorageClass "ibm-object-storage-standard-s3fs": rpc error: code = PermissionDenied desc = unable to create the bucket: StorageBucketWithUppercase
  1. Incorrect Handling of BucketVersioning: Passing Wrong Value
    `secret.yaml'
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-truen-secret
  namespace: default
data:
  bucketName: cHNhLXNlY3JldC10cnVl
  accessKey: xxx
  secretKey: yyy
  BucketVersioning: ZW5hYmxl  # enable 
stringData:
  mountOptions: |

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-bucketss-pvc-1
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-truen-secret"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml
secret/cos-s3fs-truen-secret created

% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml
persistentvolumeclaim/cos-s3fs-bucketss-pvc-1  created

% kubectl get pvc                                         
NAME                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-bucketss-pvc-1   Pending                                                                        ibm-object-storage-standard-s3fs   <unset>                 3s

% kubectl describe pvc cos-s3fs-bucketss-pvc-1                     
Name:          cos-s3fs-bucketss-pvc-1
Namespace:     default
StorageClass:  ibm-object-storage-standard-s3fs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   cos.csi.driver/secret: cos-s3fs-truen-secret
               cos.csi.driver/secret-namespace: default
               volume.beta.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age               From                                                                                               Message
  ----     ------                ----              ----                                                                                               -------
  Normal   Provisioning          4s (x4 over 11s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  External provisioner is provisioning volume for claim "default/cos-s3fs-bucketss-pvc-1"
  Warning  ProvisioningFailed    4s (x4 over 11s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  failed to provision volume with StorageClass "ibm-object-storage-standard-s3fs": rpc error: code = InvalidArgument desc = Invalid BucketVersioning value in secret: cos-s3fs-truen-secret. Value set enable. Must be 'true' or 'false'
  Normal   ExternalProvisioning  0s (x2 over 11s)  persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'cos.s3.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

@Prachi03510
Copy link
Author

Prachi03510 commented May 21, 2025

Versioning enabled via secret for volume - rclone

Positive Test Scenarios:

  1. Successfully created PVC with versioning enabled.
    cos-s3-csi-rclone-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-rclone-secret
  namespace: default
data:
  bucketName: cHJhY2hpLWRldnRlc3QwMS1idWNrZXQyMzk=
  accessKey: xxx
  secretKey: yyy
  versioningEnabled: dHJ1ZQ==
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-rclone-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-rclone-pvc
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-secret3"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml
persistentvolumeclaim/cos-rclone-pvc created

% kubectl get pvc                                                 
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc            Bound     pvc-e0fd3404-79b3-4a98-863b-ed46aac90f86   256Mi      RWO            ibm-object-storage-smart-rclone    <unset>                 4s
image
  1. Successfully created PVC with versioning disabled.
    cos-s3-csi-s3fs-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-secret1
  namespace: default
data:
  bucketName: cHJhLXNlY3Jlbi1idWNrZXRuYW1l
  accessKey: xxx
  secretKey: yyy
  versioningEnabled: ZmFsc2U=
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-pvc2
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-secret1"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret1 created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml
persistentvolumeclaim/cos-rclone-pvc2 created

% kubectl get pvc                                                    
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc2           Bound     pvc-43497d8f-0281-4e47-8ba4-f7b555bbdc7b   256Mi      RWO            ibm-object-storage-smart-rclone    <unset>                 3s
image
  1. Successfully updated PVC to enable versioning for an existing bucket.
    secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-secret2
  namespace: default
data:
  bucketName: cHNhLWRpc2FibGUtcmNsb25lYnVja2V0
  accessKey: xxx
  secretKey: yyy
  BucketVersioning: dHJ1ZQ==
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-pvc3
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-secret2"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret2 created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml   
persistentvolumeclaim/cos-rclone-pvc3 created

% kubectl get pvc                                                    
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc3           Bound     pvc-b45063ab-605e-4132-beb0-67fd9296046b   256Mi      RWO            ibm-object-storage-smart-rclone    <unset>                 3s
image
  1. Successfully updated PVC to disable versioning for an existing bucket.
    cos-s3-csi-s3fs-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-secret4
  namespace: default
data:
  bucketName: cHNhLXJjbG9uZS1idWNrZXQ=
  accessKey: xxx
  secretKey: yyy
  versioningEnabled: ZmFsc2U=
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-pvc4
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-secret4"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret4 created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml   
persistentvolumeclaim/cos-rclone-pvc4 created

% kubectl get pvc                                                    
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc4           Bound     pvc-6f00f994-f4cc-49d5-afa8-90eefaf9cc96   256Mi      RWO            ibm-object-storage-smart-rclone    <unset>                 3s
image

@Prachi03510
Copy link
Author

Negative Test Scenarios:

  1. Invalid secret name
    cos-s3-csi-rclone-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-rclone-secret-1
data:
  # Bucket name: echo -n "nkcode-devtest01" | base64
  bucketName: cHNhLWRpc2FibGUtcmNsb25lYnVja2V0
  accessKey: ZZZ
  secretKey: NNN
  bucketVersioning: ZmFsc2U=
stringData:
  mountOptions: |

cos-s3-csi-rclone-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-rclone-pvc-1
  namespace: default
  annotations:
    cos.csi.driver/secret: " cos-rclone-secret-1"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-smart-rclone
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret-1 created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml 
persistentvolumeclaim/cos-rclone-pvc-1 created

% kubectl get pvc
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc-1          Pending                                                                        ibm-object-storage-smart-rclone    <unset>                 7s

% kubectl describe pvc cos-rclone-pvc-1 
Name:          cos-rclone-pvc-1
Namespace:     default
StorageClass:  ibm-object-storage-smart-rclone
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   cos.csi.driver/secret:  cos-rclone-secret-1
               cos.csi.driver/secret-namespace: default
               volume.beta.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age                From                                                                                               Message
  ----     ------                ----               ----                                                                                               -------
  Normal   Provisioning          28s (x6 over 59s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  External provisioner is provisioning volume for claim "default/cos-rclone-pvc-1"
  Warning  ProvisioningFailed    28s (x6 over 59s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  failed to provision volume with StorageClass "ibm-object-storage-smart-rclone": "${pvc.annotations['cos.csi.driver/secret']}" resolved to " cos-rclone-secret-1" which is not a valid secret name
  Normal   ExternalProvisioning  4s (x5 over 59s)   persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'cos.s3.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  1. Invalid Bucket Name (uppercase not allowed)
    cos-s3-csi-rclone-secret.yaml
apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-rclone-secret-2
data:
  # Bucket name: echo -n "nkcode-devtest01" | base64
  bucketName: U3RvcmFnZUJ1Y2tldFdpdGhVcHBlcmNhc2U=
  accessKey: ZZZ
  secretKey: NNN
  bucketVersioning: ZmFsc2U=
stringData:
  mountOptions: |

cos-s3-csi-rclone-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-rclone-pvc-2
  namespace: default
  annotations:
    cos.csi.driver/secret: " cos-rclone-secret-2"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-smart-rclone
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret-2 created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml   
persistentvolumeclaim/cos-rclone-pvc-2 created

% kubectl get pvc                                                    
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc-2          Pending                                                                        ibm-object-storage-smart-rclone    <unset>                 3s

% kubectl describe pvc cos-rclone-pvc-2                           
Name:          cos-rclone-pvc-2
Namespace:     default
StorageClass:  ibm-object-storage-smart-rclone
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   cos.csi.driver/secret: cos-rclone-secret-2
               cos.csi.driver/secret-namespace: default
               volume.beta.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age              From                                                                                               Message
  ----     ------                ----             ----                                                                                               -------
  Normal   ExternalProvisioning  8s               persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'cos.s3.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          4s (x3 over 8s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  External provisioner is provisioning volume for claim "default/cos-rclone-pvc-2"
  Warning  ProvisioningFailed    3s (x3 over 7s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-xz5n6_d2ea50e6-333a-430b-9ef8-6ecf79cc151f  failed to provision volume with StorageClass "ibm-object-storage-smart-rclone": rpc error: code = PermissionDenied desc = unable to create the bucket: StorageBucketWithUppercase

@Prachi03510
Copy link
Author

Prachi03510 commented May 21, 2025

Versioning enabled via storage for volume - s3fs

Positive Test Scenarios:
1.Successfully created PVC with versioning enabled.
cos-s3fs-standard-sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibm-object-storage-standard-s3fs
  labels:
    ibm.cos.storageclass/tier: standard
provisioner: cos.s3.csi.ibm.io
mountOptions:
  - "multipart_size=62"
  - "max_dirty_data=51200"
  - "parallel_count=8"
  - "max_stat_cache_size=100000"
  - "retries=5"
  - "kernel_cache"
parameters:
  mounter: "s3fs"
  client: "awss3"
  cosEndpoint: "https://s3.direct.us-west.cloud-object-storage.appdomain.cloud"
  locationConstraint: "us-west-standard"
  bucketVersioning: "true"  
  csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['cos.csi.driver/secret']}
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
reclaimPolicy: Delete

cos-s3-csi-s3fs-secret.yaml

apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-bucket-secret-s
  namespace: default
data:
  bucketName: cHNhLXN0LWJ1Y2tldA==
  accessKey: xxx
  secretKey: yyy
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-bucket-pvc-s
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-bucket-secret-s"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml  
secret/cos-s3fs-bucket-secret-s created

% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml   
persistentvolumeclaim/cos-s3fs-bucketss-pvc-s created

% kubectl  get pvc
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-bucketss-pvc-s   Bound     pvc-9bb6fedb-bb40-4603-95cd-c7b8c5869eab   256Mi      RWO            ibm-object-storage-standard-s3fs   <unset>                 6s

% kubectl describe sc ibm-object-storage-standard-s3fs 
Name:            ibm-object-storage-standard-s3fs
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"kustomize","app.kubernetes.io/part-of":"ibm-object-csi-driver","ibm.cos.storageclass/tier":"standard"},"name":"ibm-object-storage-standard-s3fs"},"mountOptions":["multipart_size=62","max_dirty_data=51200","parallel_count=8","max_stat_cache_size=100000","retries=5","kernel_cache"],"parameters":{"bucketVersioning":"true","client":"awss3","cosEndpoint":"https://s3.direct.us-south.cloud-object-storage.appdomain.cloud","csi.storage.k8s.io/node-publish-secret-name":"${pvc.annotations['cos.csi.driver/secret']}","csi.storage.k8s.io/node-publish-secret-namespace":"${pvc.namespace}","locationConstraint":"us-south-standard","mounter":"s3fs"},"provisioner":"cos.s3.csi.ibm.io","reclaimPolicy":"Delete"}

Provisioner:           cos.s3.csi.ibm.io
Parameters:            bucketVersioning=true,client=awss3,cosEndpoint=https://s3.direct.us-south.cloud-object-storage.appdomain.cloud,csi.storage.k8s.io/node-publish-secret-name=${pvc.annotations['cos.csi.driver/secret']},csi.storage.k8s.io/node-publish-secret-namespace=${pvc.namespace},locationConstraint=us-south-standard,mounter=s3fs
AllowVolumeExpansion:  <unset>
MountOptions:
  multipart_size=62
  max_dirty_data=51200
  parallel_count=8
  max_stat_cache_size=100000
  retries=5
  kernel_cache
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>
image
  1. Successfully created PVC with versioning disabled.
    cos-s3fs-standard-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibm-object-storage-standard-s3fs
  labels:
    ibm.cos.storageclass/tier: standard
provisioner: cos.s3.csi.ibm.io
mountOptions:
  - "multipart_size=62"
  - "max_dirty_data=51200"
  - "parallel_count=8"
  - "max_stat_cache_size=100000"
  - "retries=5"
  - "kernel_cache" 
parameters:
  mounter: "s3fs"
  client: "awss3"
  cosEndpoint: "https://s3.direct.us-west.cloud-object-storage.appdomain.cloud"
  locationConstraint: "us-west-standard"
  bucketVersioning: "false"  
  csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['cos.csi.driver/secret']}
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
reclaimPolicy: Delete

cos-s3-csi-s3fs-secret.yaml

apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-s3fs-bucket-secret-s2
  namespace: default
data:
  bucketName: cHNhLXN0LWJ1Y2tldA==
  accessKey: xxx
  secretKey: yyy
stringData:
  # uid: "3000" # Provide uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
  mountOptions: |

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-bucketss-pvc-s2
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-bucket-secret-s2"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-secret.yaml
secret/cos-s3fs-bucket-secret-s2 created

% kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml   
persistentvolumeclaim/cos-s3fs-bucketss-pvc-s2 created

% kubectl  get pvc                                                 
NAME                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-bucketss-pvc-s    Bound     pvc-9bb6fedb-bb40-4603-95cd-c7b8c5869eab   256Mi      RWO            ibm-object-storage-standard-s3fs   <unset>                 5m

% kubectl describe sc ibm-object-storage-standard-s3fs               
Name:            ibm-object-storage-standard-s3fs
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"kustomize","app.kubernetes.io/part-of":"ibm-object-csi-driver","ibm.cos.storageclass/tier":"standard"},"name":"ibm-object-storage-standard-s3fs"},"mountOptions":["multipart_size=62","max_dirty_data=51200","parallel_count=8","max_stat_cache_size=100000","retries=5","kernel_cache"],"parameters":{"bucketVersioning":"false","client":"awss3","cosEndpoint":"https://s3.direct.us-south.cloud-object-storage.appdomain.cloud","csi.storage.k8s.io/node-publish-secret-name":"${pvc.annotations['cos.csi.driver/secret']}","csi.storage.k8s.io/node-publish-secret-namespace":"${pvc.namespace}","locationConstraint":"us-south-standard","mounter":"s3fs"},"provisioner":"cos.s3.csi.ibm.io","reclaimPolicy":"Delete"}

Provisioner:           cos.s3.csi.ibm.io
Parameters:            bucketVersioning=false,client=awss3,cosEndpoint=https://s3.direct.us-south.cloud-object-storage.appdomain.cloud,csi.storage.k8s.io/node-publish-secret-name=${pvc.annotations['cos.csi.driver/secret']},csi.storage.k8s.io/node-publish-secret-namespace=${pvc.namespace},locationConstraint=us-south-standard,mounter=s3fs
AllowVolumeExpansion:  <unset>
MountOptions:
  multipart_size=62
  max_dirty_data=51200
  parallel_count=8
  max_stat_cache_size=100000
  retries=5
  kernel_cache
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>
image

@Prachi03510
Copy link
Author

Prachi03510 commented May 21, 2025

Versioning enabled via storage for volume - rclone

Positive Test Scenarios:

  1. Successfully created PVC with versioning enabled.
    cos-s3fs-standard-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibm-object-storage-smart-rclone
  labels:
    ibm.cos.storageclass/tier: smart
provisioner: cos.s3.csi.ibm.io
mountOptions:
    - "acl=private"
    - "bucket_acl=private"
    - "upload_cutoff=256Mi"
    - "chunk_size=64Mi"
    - "max_upload_parts=64"
    - "upload_concurrency=20"
    - "copy_cutoff=1Gi"
    - "memory_pool_flush_time=30s"
    - "disable_checksum=true"
parameters:
  mounter: "rclone"
  client: "awss3"
  cosEndpoint: "https://s3.direct.us-west.cloud-object-storage.appdomain.cloud"
  locationConstraint: "us-west-smart"
  bucketVersioning: "true"  
  csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['cos.csi.driver/secret']}
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
reclaimPolicy: Delete

cos-s3-csi-rclone-secret.yaml

apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-rclone-secret-r2
data:
  bucketName: cC1yY2xvbmUtZW5hYmxlLWJ1Y2tldA==
  accessKey: xxx
  secretKey: yyy
  # echo -n <kpRootKeyCRN> | base64
  # kpRootKeyCRN: # base64 encoded Key Protect Root key CRN
stringData:
  mountOptions: |

cos-s3-csi-rclone-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-rclone-pvc-r3
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-rclone-secret-r2"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-smart-rclone
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret-r created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml   
persistentvolumeclaim/cos-rclone-pvc-r created

% kubectl  get pvc                                                   
NAME                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc-r           Bound     pvc-c0615c31-304a-4c60-83b2-bbb8b3514604   256Mi      RWO            ibm-object-storage-smart-rclone    <unset>                 14s
image
  1. Successfully created PVC with versioning disabled.
    cos-s3fs-standard-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibm-object-storage-smart-rclone
  labels:
    ibm.cos.storageclass/tier: smart
provisioner: cos.s3.csi.ibm.io
mountOptions:
    - "acl=private"
    - "bucket_acl=private"
    - "upload_cutoff=256Mi"
    - "chunk_size=64Mi"
    - "max_upload_parts=64"
    - "upload_concurrency=20"
    - "copy_cutoff=1Gi"
    - "memory_pool_flush_time=30s"
    - "disable_checksum=true"
parameters:
  mounter: "rclone"
  client: "awss3"
  cosEndpoint: "https://s3.direct.us-west.cloud-object-storage.appdomain.cloud"
  locationConstraint: "us-west-smart"
  bucketVersioning: "false"  
  csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['cos.csi.driver/secret']}
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
reclaimPolicy: Delete

cos-s3-csi-rclone-secret.yaml

apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
  name: cos-rclone-secret-r
data:
  bucketName: cHNhLXN0LXJjbG9uZS1idWNrZXQ=
  accessKey: xxx
  secretKey: yyy
  # echo -n <kpRootKeyCRN> | base64
  # kpRootKeyCRN: # base64 encoded Key Protect Root key CRN
stringData:
  mountOptions: |

cos-s3-csi-rclone-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-rclone-pvc-r
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-rclone-secret-r"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-smart-rclone
% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-secret.yaml
secret/cos-rclone-secret-r2 created

% kubectl create -f examples/kubernetes/cos-s3-csi-rclone-pvc.yaml   
persistentvolumeclaim/cos-rclone-pvc-r3 created

% kubectl  get pvc                                                   
NAME                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-rclone-pvc-r3          Bound     pvc-ffdb124b-0217-4216-9aba-8e072383d6de   256Mi      RWO            ibm-object-storage-smart-rclone    <unset>                 4s
image

@Prachi03510
Copy link
Author

Negative Scenario

  1. Set Versioning on a Non-Existent Secret
    cos-s3fs-standard-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibm-object-storage-standard-s3fs
  labels:
    ibm.cos.storageclass/tier: standard
provisioner: cos.s3.csi.ibm.io
mountOptions:
  - "multipart_size=62"
  - "max_dirty_data=51200"
  - "parallel_count=8"
  - "max_stat_cache_size=100000"
  - "retries=5"
  - "kernel_cache"
parameters:
  mounter: "s3fs"
  client: "awss3"
  cosEndpoint: "https://s3.direct.us-west.cloud-object-storage.appdomain.cloud"
  locationConstraint: "us-west-standard"
  BucketVersioning: "true"  
  csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['cos.csi.driver/secret']}
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
reclaimPolicy: Delete

cos-s3-csi-s3fs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-s3fs-non-pvc
  namespace: default
  annotations:
    cos.csi.driver/secret: "cos-s3fs-nonexisting-secret"
    cos.csi.driver/secret-namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: ibm-object-storage-standard-s3fs
prachishivanandanure@Prachis-MacBook-Pro ibm-object-csi-driver % kubectl create -f examples/kubernetes/cos-s3-csi-s3fs-pvc.yaml   
persistentvolumeclaim/cos-s3fs-non-pvc created
prachishivanandanure@Prachis-MacBook-Pro ibm-object-csi-driver % kubectl get pvc
NAME                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                       VOLUMEATTRIBUTESCLASS   AGE
cos-s3fs-non-pvc           Pending                                                                        ibm-object-storage-standard-s3fs   <unset>                 6s

% kubectl describe pvc cos-s3fs-non-pvc 
Name:          cos-s3fs-non-pvc
Namespace:     default
StorageClass:  ibm-object-storage-standard-s3fs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   cos.csi.driver/secret: cos-s3fs-nonexisting-secret
               cos.csi.driver/secret-namespace: default
               volume.beta.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: cos.s3.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age               From                                                                                               Message
  ----     ------                ----              ----                                                                                               -------
  Normal   ExternalProvisioning  2s (x6 over 65s)  persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'cos.s3.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          1s (x7 over 65s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-mbbgx_263d176f-dc5a-4842-aa15-cdcca7626ed4  External provisioner is provisioning volume for claim "default/cos-s3fs-non-pvc"
  Warning  ProvisioningFailed    1s (x7 over 65s)  cos.s3.csi.ibm.io_ibm-object-csi-controller-5fbcf6874c-mbbgx_263d176f-dc5a-4842-aa15-cdcca7626ed4  failed to provision volume with StorageClass "ibm-object-storage-standard-s3fs": rpc error: code = InvalidArgument desc = Secret resource not found error getting Secret: secrets "cos-s3fs-nonexisting-secret" not found

@@ -182,6 +185,26 @@ func (s *COSSession) DeleteBucket(bucket string) error {
return err
}

func (s *COSSession) SetBucketVersioning(bucket string, enable bool) error {
status := "Suspended"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets use constant provided by s3 package

	status = s3.BucketVersioningStatusSuspended
	if enable {
		status = s3.BucketVersioningStatusEnabled
	}

@Prachi03510
Copy link
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants