-
Notifications
You must be signed in to change notification settings - Fork 1.8k
OSDOCS-13459#Add cross-subscription support for Azure File #92108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
lpettyjo
wants to merge
1
commit into
openshift:main
Choose a base branch
from
lpettyjo:OSDOCS-13459
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
87 changes: 87 additions & 0 deletions
87
...storage-csi-azure-file-cross-sub-dynamic-pre-provisioning-pv-pvc-procedure.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| // * storage/container_storage_interface/persistent_storage-csi-azure-file.adoc | ||
| // | ||
| :_mod-docs-content-type: PROCEDURE | ||
| [id="persistent-storage-csi-azure-file-cross-sub-dynamic-pre-provisioning-pv-pvc-procedure_{context}"] | ||
| = Static provisioning across subscriptions for Azure File by creating a PV and PVC: | ||
|
|
||
| .Prerequisites | ||
| * Installed {product-title} cluster on Azure with the service principal or managed identity as an Azure identity in one subscription (call it Subscription A) | ||
|
|
||
| * Access to another subscription (call it Subscription B) with the storage that is in the same tenant as the cluster | ||
|
|
||
| * Logged in to the Azure CLI | ||
|
|
||
| .Procedure | ||
| . For your Azure File share, record the resource group, storage account, storage account key, and Azure File name. These values are used for the next steps. | ||
|
|
||
| . Create a secret for the persistent volume parameter `spec.csi.nodeStageSecretRef.name` by running the following command: | ||
| + | ||
| [source, terminal] | ||
| ---- | ||
| $ oc create secret generic azure-storage-account-<storageaccount-name>-secret --from-literal=azurestorageaccountname="<azure-storage-account-name>" --from-literal azurestorageaccountkey="<azure-storage-account-key>" --type=Opaque | ||
| ---- | ||
| + | ||
| Where: | ||
| `<azure-storage-account-name>` and `<azure-storage-account-key>` are the Azure storage account name and key respectively that you recorded in Step 1. | ||
|
|
||
| . Create a persistent volume (PV) by using a similar configuration to the following example file: | ||
| + | ||
| .Example PV YAML file | ||
| [source,terminal] | ||
| ---- | ||
| apiVersion: v1 | ||
| kind: PersistentVolume | ||
| metadata: | ||
| annotations: | ||
| pv.kubernetes.io/provisioned-by: file.csi.azure.com | ||
| name: <pv-name> <1> | ||
| spec: | ||
| capacity: | ||
| storage: 10Gi <2> | ||
| accessModes: | ||
| - ReadWriteMany | ||
| persistentVolumeReclaimPolicy: Retain | ||
| storageClassName: <sc-name> <3> | ||
| mountOptions: | ||
| - cache=strict | ||
| - nosharesock | ||
| - actimeo=30 | ||
| - nobrl | ||
| csi: | ||
| driver: file.csi.azure.com | ||
| volumeHandle: "{resource-group-name}#{storage-account-name}#{file-share-name}" <4> | ||
| volumeAttributes: | ||
| shareName: <existing-file-share-name> <5> | ||
| nodeStageSecretRef: | ||
| name: <secret-name> <6> | ||
| namespace: <secret-namespace> <7> | ||
| ---- | ||
| <1> The name of the PV. | ||
| <2> The size of the PV. | ||
| <3> The storage class name. | ||
| <4> Ensure that `volumeHandle` is unique for every identical share in the cluster. | ||
| <5> For `<existing-file-share-name>, use only the file share name and not the full path. | ||
| <6> The secret name created in the previous step. | ||
| <7> The namespace where the secret resides. | ||
|
|
||
| . Create a persistent value claim (PVC) specifying the existing Azure File share referenced in Step 1 using a similar configuration to the following: | ||
| + | ||
| .Example PVC YAML file | ||
| [source,yaml] | ||
| ---- | ||
| apiVersion: v1 | ||
| kind: PersistentVolumeClaim | ||
| metadata: | ||
| name: <pvc-name> <1> | ||
| spec: | ||
| storageClassName: <sc-name> <2> | ||
| accessModes: | ||
| - ReadWriteMany | ||
| resources: | ||
| requests: | ||
| storage: 5Gi | ||
| ---- | ||
| <1> The name of the PVC. | ||
| <2> The name of the storage class that you specified for the PV in the previous step. | ||
131 changes: 131 additions & 0 deletions
131
...persistent-storage-csi-azure-file-cross-sub-dynamic-provisioning-procedure.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| // * storage/container_storage_interface/persistent_storage-csi-azure-file.adoc | ||
| // | ||
| :_mod-docs-content-type: PROCEDURE | ||
| [id="persistent-storage-csi-azure-file-cross-sub-dynamic-provisioning-procedure_{context}"] | ||
| = Dynamic provisioning across subscriptions for Azure File | ||
|
|
||
| .Prerequisites | ||
| * Installed {product-title} cluster on Azure with the service principal or managed identity as an Azure identity in one subscription (call it Subscription A) | ||
|
|
||
| * Access to another subscription (call it Subscription B) with the storage that is in the same tenant as the cluster | ||
|
|
||
| * Logged in to the Azure CLI | ||
|
|
||
| .Procedure | ||
| To use Azure File dynamic provisioning across subscriptions: | ||
|
|
||
| . Record the Azure identity (service principal or managed identity) by running the following applicable commands. The Azure identity is needed in a later step: | ||
| + | ||
| * If using the _service principal_ as the Azure identity when installing the cluster: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ sp_id=$(oc -n openshift-cluster-csi-drivers get secret azure-file-credentials -o jsonpath='{.data.azure_client_id}' | base64 --decode) | ||
|
|
||
| $ az ad sp show --id ${sp_id} --query displayName --output tsv | ||
| ---- | ||
| + | ||
| * If using the _managed identity_ as the Azure identity when installing the cluster: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ mi_id=$(oc -n openshift-cluster-csi-drivers get secret azure-file-credentials -o jsonpath='{.data.azure_client_id}' | base64 --decode) | ||
|
|
||
| $ az identity list --query "[?clientId=='${mi_id}'].{Name:name}" --output tsv | ||
| ---- | ||
|
|
||
| . Grant the Azure identity (service principal or managed identity) permission to access the resource group in another Subscription B where you want to provision the Azure File share by doing one of the following: | ||
|
|
||
| * Run the following Azure CLI command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| az role assignment create \ | ||
| --assignee <object-id-or-app-id> \ | ||
| --role <role-name> \ | ||
| --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name> | ||
| ---- | ||
| + | ||
| Where: | ||
| + | ||
| `<object-id-or-app-id>`: The service principal or managed identity that you obtained from the previous step, such as `sp_id` or `mi_id`. | ||
| + | ||
| `<role-name>`: Role name. Contributor or your own role with required permissions. | ||
| + | ||
| `<subscription-id>`: Subscription B ID. | ||
| + | ||
| `<resource-group-name>`: Subscription B resource group name. | ||
| + | ||
| Or | ||
| + | ||
| * Log in to the Azure portal and on the left menu, click *Resource groups*: | ||
|
|
||
| .. Choose the resource group in Subscription B to which you want to assign a role by clicking *resource group* -> *Access control (IAM)* -> *Role assignments* tab to view current assignments, and then click *Add* > *Add role assignment*. | ||
|
|
||
| .. On the *Role* tab, choose the contributor role to assign, and then click *Next*. You can also create and choose your own role with required permission. | ||
|
|
||
| .. On the *Members* tab: | ||
| ... Choose an assignee by selecting the type of assignee: user, group, or service principal (or managed identity). | ||
| ... Click *Select members*. | ||
| ... Search for, and then select the desired service principal or managed identity recorded in the previous step. | ||
| ... Click *Select* to confirm. | ||
| .. On the *Review + assign* tab, review the settings. | ||
|
|
||
| .. To finish the role assignment, click *Review + assign*. | ||
| + | ||
lpettyjo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| [NOTE] | ||
| ==== | ||
| If you only want to use a specific storage account to provision the Azure File share, you can also obtain the Azure identity (service principal or managed identity) permission to access the storage account by using similar steps. | ||
| ==== | ||
|
|
||
| . Create an Azure File storage class by using a similar configuration to the following: | ||
| + | ||
| .Example Azure File storage class YAML file | ||
| [source, yaml] | ||
| ---- | ||
| allowVolumeExpansion: true | ||
| apiVersion: storage.k8s.io/v1 | ||
| kind: StorageClass | ||
| metadata: | ||
| name: <sc-name> <1> | ||
| mount options: | ||
| - mfsymlinks | ||
| - cache=strict | ||
| - nosharesock | ||
| - actimeo=30 | ||
| parameters: | ||
| subscriptionID: <xxxx-xxxx-xxxx-xxxx-xxxx> <2> | ||
| resourceGroup: <resource group name> <3> | ||
| storageAccount: <storage account> <4> | ||
| skuName: <skuName> <5> | ||
| provisioner: file.csi.azure.com | ||
| reclaimPolicy: Delete | ||
| volumeBindingMode: Immediate | ||
| ---- | ||
| <1> The name of the storage class | ||
| <2> The subscription B ID | ||
| <3> The Subscription B resource group name | ||
| <4> The storage account name, if you want to specify your own | ||
| <5> The name of the SKU type | ||
|
|
||
| . Create a persistent volume claim (PVC) that specifies the Azure File storage class that you created in the previous step by using a similar configuration to the following: | ||
| + | ||
| .Example PVC YAML file | ||
| [source, yaml] | ||
| ---- | ||
| apiVersion: v1 | ||
| kind: PersistentVolumeClaim | ||
| metadata: | ||
| name: <pvc-name> <1> | ||
| spec: | ||
| storageClassName: <sc-name-cross-sub> <2> | ||
| accessModes: | ||
| - ReadWriteMany | ||
| resources: | ||
| requests: | ||
| storage: 5Gi | ||
| ---- | ||
| <1> The name of the PVC. | ||
| <2> The name of the storage class that you created in the previous step. | ||
15 changes: 15 additions & 0 deletions
15
modules/persistent-storage-csi-azure-file-cross-sub-overview.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| // * storage/container_storage_interface/persistent_storage-csi-azure-file.adoc | ||
| // | ||
|
|
||
| :_mod-docs-content-type: CONCEPT | ||
| [id="persistent-storage-csi-azure-file-cross-sub-overview_{context}"] | ||
| = Azure File cross-subscription support | ||
|
|
||
| Cross-subscription support allows you to have an {product-title} cluster in one Azure subscription and mount your Azure file share in another Azure subscription by using the Azure File Container Storage Interface (CSI) driver. | ||
|
|
||
| [IMPORTANT] | ||
| ==== | ||
| Both the {product-title} cluster and the Azure File share (pre-provisioning or to be provisioned) should be inside the same tenant. | ||
| ==== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In case of static provisioning the SC name does not really matter it does not need to actually exists right? Should we call it out explicitly to avoid confusion ? i.e avoid customers wondering if they should create one.
@duanwei33
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it matters, the sc name in PV and PVC helps binding, it could help avoid unexpected behaviour. (Like there are other azure-disk PVs without sc name and the size is the same)
But yes, the storage class itself (with the same name) exists or not will not impact the binding, I think for customer scenario, non-admin users have no access to list PV, so usually an admin will create a sc which a meaningful name and create pvs with the same sc name inside, then the non-admin users create the PVC with the sc for expected pvs.
In short, let's keep sc name in both pvc and pv.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok to keep it but it can be misleading as customers will not know what to set here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so.....leave as is or change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was wondering if we should mention that the SC does not need to exist.
What happens if a user sets an existing (unrelated) storage class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personally think we should keep the SC in the example + comment that it's used to disable dynamic provisioning for such PV / PVC.
In theory nothing, the PVC will try to bind to an existing PV. In practice, when the manually created PV and PVC accidentally don't match, Kubernetes will try to dynamically provision a new PV. Which is not what the user wants. Either a dedicated SC with
provisioner: kubernetes.io/no-provisioneror non-existing SC will ensure the dynamic provisioning does not happen. Mis-matched PV + PVC stay Pending and user can fix their errors.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree to keep the SC in the PVC/PV definition i was just wondering if we could add an extra note on the fact that the SC should not actually to avoid potential user errors (or as Jan said create a no-provisioner one). Not mentioning anything can be error prone i think