Skip to content

Usage.md

Latest commit

 

History

History
314 lines (238 loc) · 11.6 KB

Usage.md

File metadata and controls

314 lines (238 loc) · 11.6 KB

Scality COSI Driver Usage Guide: Bucket Provisioning & Access Control

This document provides an overview and step-by-step guidance for implementing Bucket Provisioning (both Greenfield and Brownfield) and Access Control using the Scality COSI Driver. Example YAML manifests can be found in the cosi-examples folder.

Note

The Scality COSI Driver supports standard AWS S3 and IAM compliant storage solutions like Scality RING, Scality ARTESCA and AWS S3 & IAM

Prerequisites

Before proceeding, ensure that the following components are installed on your cluster:

  1. Kubernetes and Helm: Ensure your Kubernetes cluster is properly configured, and Helm v3+ is installed. The COSI specification was introduced in Kubernetes 1.25. We recommend using one of the latest supported Kubernetes versions.
  2. Kubernetes Container Object Storage Interface (COSI) CRDs
  3. Container Object Storage Interface Controller

Refer to the quick start guide in the README for installation instructions.

Common Setup Steps

  1. Create the IAM User (by the Storage Administrator)

    Create an IAM user and a pair of Access Key ID and Secret Access Key. This user will be used by COSI driver. Assign S3/IAM permissions that allow bucket creation and user management. Permissions needed by COSI driver:

    • S3:CreateBucket
    • S3:DeleteBucket
    • IAM:GetUser
    • IAM:CreateUser
    • IAM:DeleteUser
    • IAM:PutUserPolicy
    • IAM:DeleteUserPolicy
    • IAM:ListAccessKeys
    • IAM:CreateAccessKey
    • IAM:DeleteAccessKey

  2. Collect Access & Endpoint Details

    The Storage Administrator provides the below details to the Kubernetes Administrator:

    • S3 endpoint (and IAM endpoint, if different)
    • Region
    • Access Key ID & Secret Key
    • tlsCert, if needed. COSI driver uses the the AWS setting InsecureSkipVerify for HTTPs endpoints if not provided.

  3. Create a Kubernetes Secret (by the Kubernetes Administrator)

    The Kubernetes Administrator creates a secret containing the above credentials and configuration details:

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: s3-secret-for-cosi
type: Opaque
stringData:
  accessKeyId: <ACCESS_KEY_ID>
  secretAccessKey: <SECRET_ACCESS_KEY>
  endpoint: <S3_ENDPOINT>
  region: <REGION>
  iamEndpoint: <OPTIONAL_IAM_ENDPOINT>
  tlsCert: |-
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
EOF

    [!NOTE] Update <ACCESS_KEY_ID>, <SECRET_ACCESS_KEY>, <S3_ENDPOINT>, <REGION>, with valid values for your environment. If your endpoint does not require a TLS cert, you can remove it. Similarly, add IAM endpoint with iamEndpoint if its different from S3 endpoint otherwise remove it. If using TLS cert, include the certificate content (PEM-encoded) in the stringData section of the Secret. Use a multi-line block scalar (|-) in YAML so that the certificate (with newlines) is preserved correctly. For HTTPS endpoint if not using TLS certificate, COSI driver will use the InsecureSkipVerify flag.

1. Bucket Provisioning

In the Scality COSI Driver, both Greenfield and Brownfield provisioning share similar steps, with minor differences in how resources (Bucket, BucketClaim) are created.

Note: For fully working examples, see the YAMLs in the cosi-examples/brownfield and cosi-examples/greenfield directories.

1.1 Greenfield: Creating a New Bucket

Greenfield provisioning will create a brand-new S3 bucket in your object store, managed by Kubernetes. Examples can be found here.

  1. Create a BucketClass A BucketClass defines how buckets should be provisioned or deleted. The bucket class name is used as a prefix for bucket name by COSI:

    cat <<EOF | kubectl apply -f -
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClass
metadata:
  name: greenfield-bucketclass
driverName: cosi.scality.com
deletionPolicy: Delete
parameters:
  objectStorageSecretName: s3-secret-for-cosi
  objectStorageSecretNamespace: default
EOF
    • driverName must match the Scality driver (default: cosi.scality.com). This determined by driver-prefix in the COSI driver deployment. For more information check COSI Driver Parameters.
    • deletionPolicy can be Delete or Retain.
    • objectStorageSecretName and objectStorageSecretNamespace reference the secret you created earlier in Common Setup Steps. For more information check COSI Driver Parameters.

  2. Create a BucketClaim

    A BucketClaim requests a new bucket using the above BucketClass:

    cat <<EOF | kubectl apply -f -
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClaim
metadata:
  name: my-greenfield-bucketclaim
spec:
  bucketClassName: greenfield-bucketclass
  protocols:
    - S3
EOF
    • This will automatically provision a new S3 bucket in the backend using the COSI driver. The COSI driver will use credentials and endpoint mentioned in the secret specified in the BucketClass to create the bucket.
    • The actual bucket name on S3 is typically generated by the driver (e.g., greenfield-bucketclassf5e6461d-35c9-4fbe-b679-cf391a223d96 in the format <bucketclassName><UUID>).
    • Only S3 protocol is supported at the moment.

1.2 Brownfield: Using an Existing Bucket

Brownfield provisioning allows you to manage an already-existing S3 bucket in Kubernetes.

  1. Verify Existing Bucket

    Ensure the bucket already exists in S3 either through Storage Administrator or by running the following AWS CLI command:

    aws s3api head-bucket --bucket <EXISTING_BUCKET_NAME> --endpoint-url <S3_ENDPOINT>

  2. Create a BucketClass

    Similar to Greenfield, but you will typically still reference the same secret:

    cat <<EOF | kubectl apply -f -
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClass
metadata:
  name: brownfield-bucketclass
driverName: cosi.scality.com
deletionPolicy: Delete
parameters:
  objectStorageSecretName: s3-secret-for-cosi
  objectStorageSecretNamespace: default
EOF

    [!NOTE] For Brownfield, existing buckets when imported using the steps below, do not follow the deletionPolicy even if it set to Delete . All buckets created using this bucket class for greenfield scenario will still respect the deletionPolicy

  3. Create the Bucket Resource/Instance This is where we tell Kubernetes about the existing bucket:

    cat <<EOF | kubectl apply -f -
apiVersion: objectstorage.k8s.io/v1alpha1
kind: Bucket
metadata:
  name: "<EXISTING_BUCKET_NAME>"
spec:
  bucketClaim: {}
  driverName: cosi.scality.com
  bucketClassName: brownfield-bucketclass
  driverName: cosi.scality.com
  deletionPolicy: Retain
  existingBucketID: "<EXISTING_BUCKET_NAME>"
  parameters:
    objectStorageSecretName: s3-secret-for-cosi
    objectStorageSecretNamespace: default
  protocols:
    - S3
EOF
    • name and existingBucketID should be the same as the existing bucket name in S3 storage.

  4. Create the BucketClaim Reference the existing Bucket resource/instance by name via existingBucketName:

    cat <<EOF | kubectl apply -f -
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClaim
metadata:
  name: my-brownfield-bucketclaim
spec:
  bucketClassName: brownfield-bucketclass
  existingBucketName: "<EXISTING_BUCKET_NAME>"
  protocols:
    - S3
EOF
    • existingBucketName should match the name of the Bucket resource/instance created in the previous step for Bucket Instance.

Bucket Provisioning Cleanup

To remove the buckets and associated Kubernetes resources:

  • Greenfield:

    kubectl delete bucketclaim my-greenfield-bucketclaim
    • Deleting the BucketClaim will remove the underlying bucket only if:
      • deletionPolicy was set to Delete in BucketClass.
      • The bucket is empty at the time of deletion.

  • Brownfield:

    kubectl delete bucketclaim my-brownfield-bucketclaim
    • Deleting the BucketClaim and Bucket resources in Kubernetes does not delete the actual pre-existing bucket in S3 even if deletionPolicy is Delete.

2. Access Control (Common to Greenfield & Brownfield)

Access Control configuration is effectively the same for both Greenfield and Brownfield. Once the BucketClaim is ready, you can request credentials for the bucket via a BucketAccess resource.

2.1 Create a BucketAccessClass

A BucketAccessClass defines how access (IAM policy or S3 keys) is granted:

cat <<EOF | kubectl apply -f -
kind: BucketAccessClass
apiVersion: objectstorage.k8s.io/v1alpha1
metadata:
  name: bucket-access-class
driverName: cosi.scality.com
authenticationType: KEY
parameters:
  objectStorageSecretName: s3-secret-for-cosi
  objectStorageSecretNamespace: default
EOF

Note

  • authenticationType is Key for basic S3 key-based credentials.
  • objectStorageSecretName and objectStorageSecretNamespace reference the secret you created earlier in Common Setup Steps.

2.2 Request Bucket Access

Once the BucketClaim is bound (Greenfield or Brownfield), create a BucketAccess to generate a credential secret in the cluster:

cat <<EOF | kubectl apply -f -
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketAccess
metadata:
  name: my-bucketaccess
spec:
  bucketClaimName: my-greenfield-bucketclaim  # or my-brownfield-bucketclaim
  bucketAccessClassName: bucket-access-class
  credentialsSecretName: my-s3-credentials
  protocol: S3
EOF
  • bucketClaimName references the claim from the previous section.
  • credentialsSecretName is the name of the secret that will contain the newly generated credentials (Access Key ID / Secret Key).

Once the BucketAccess is created, the driver will:

  • Create IAM user. The user name will be generated by the COSI controller as ba-$UUID.
  • Generate an inline policy with name from bucket name, for the user with the correct full S3 permission on the associated bucket.
  • Create a new Kubernetes Secret (my-s3-credentials) with the S3 Access Key ID and Secret Key of the IAM user previously created. This will follow the BucketInfo format.

2.3 Revoking Access

To revoke access for a user or application:

kubectl delete bucketaccess my-bucketaccess

This triggers the removal of the IAM user/access keys on the S3 side.

Validation & Troubleshooting

  • Bucket Verification: Confirm the bucket exists (or is deleted) in S3 as expected.

  • IAM User Verification: Check that the necessary IAM user/policies are created or removed upon BucketAccess creation/deletion.

  • Kubernetes Secret: Inspect the auto-generated credentialsSecretName to see if keys have been populated:

    kubectl get secret my-s3-credentials -o yaml

Further Reading