Skip to content

OSDOCS-12804#Increase vSphere max vols per node #91472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

OSDOCS-12804#Increase vSphere max vols per node #91472

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions modules/persistent-storage-csi-vsphere-increase-max-vols-per-node-overview.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
// Module included in the following assemblies:
//
// * storage/container_storage_interface/persistent-storage-csi-vsphere.adoc
//

:_mod-docs-content-type: CONCEPT
[id="persistent-storage-csi-vsphere-increase-max-vols-per-node-overview_{context}"]
= Increasing maximum volumes per node for vSphere

For VMware vSphere version 7, {product-title} restricts the maximum number of volumes per node to 59.

However, for vSphere version 8 or later, you can increase the allowable number of volumes per node to a maximum of 255. Otherwise, the default value remains at 59.

[IMPORTANT]
====
You must have an homogeneous vSphere 8 environment that only contains ESXi 8 hypervisors. Heterogeneous environments that contain a mix of versions of ESXi other than 8 are not allowed. In such heterogenous environment, if you set a value greater than 59, the cluster degrades.
====

.Limitations

* You must be running VMware vSphere version 8 or later.

* You can potentially exceed the limit of 2048 virtual disks per host if you increase the maximum number of volumes per node on enough nodes. This can occur because there is no Distributed Resource scheduler (DRS) validation for vSphere to ensure you do not exceed this limit.
57 changes: 57 additions & 0 deletions modules/persistent-storage-csi-vsphere-increase-max-vols-per-node-procedure.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
// Module included in the following assemblies:
//
// * storage/container_storage_interface/persistent-storage-csi-vsphere.adoc
//

:_mod-docs-content-type: PROCEDURE
[id="persistent-storage-csi-vsphere-increase-max-vols-per-node_{context}"]
= Increasing the maximum allowable volumes per node for vSphere

.Prerequisites
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As prerequisite customers need to set pvscsiCtrlr256DiskSupportEnabled to true in vCenter.
We need to document it but be VERY careful with the wording.
This is NOT officially supported by VMware and this is a global vCenter setting so even if customers want to try it on say a OCP test cluster, this setting will affect the entire vCenter which could also host production workloads.
@gnufied @RomanBednar

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lpettyjo There was a thread opened for this and then flagged as "Done", but I don't see this prerequisite note yet - PTAL.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gcharot @gnufied @RomanBednar I searched on this parameter in vSphere docs and didn't find anything, so I don't have a specific place to point users to. Maybe we need to discuss how to approach this due to its sensitive nature.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it is not documented anywhere in a public facing doc from Broadcom. That is the rub. @gcharot should we document our own stuff or ask customer to talk to Broadcom?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should mention the parameter name but clearly mention that this is not fully supported by VMware as well as pointing out this is a cluster wide option.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* Access to the {product-title} web console.

* Access to the cluster as a user with the cluster-admin role.

* Access to VMware vSphere vCenter.

* In vCenter, ensure that the parameter `pvscsiCtrlr256DiskSupportEnabled` is set to 'True'.
+
[IMPORTANT]
====
Changing the `pvscsiCtrlr256DiskSupportEnabled` parameter is not fully supported by VMware. Also, the parameter is a cluster-wide option.
====

.Procedure

Use the following procedure to increase the maximum number of volumes per node for vSphere:

. Click *Administration* > *CustomResourceDefinitions*.

. On the *CustomResourceDefinitions* page next to the *Name* dropdown box, type "clustercsidriver".

. Click *CRD ClusterCSIDriver*.

. Click the *Instances* tab.

. Click *csi.vsphere.vmware.com*.

. Click the *YAML* tab.

. Set the parameter `spec.driverConfig.driverType` to `vSphere`.

. Add the parameter `spec.driverConfig.vSphere.maxAllowedBlockVolumesPerNode` to the YAML file, and provide a value for the desired maximum number of volumes per node as in the following sample YAML file:
+
[source,yaml]
.Sample YAML file for adding the parameter maxAllowedBlockVolumesPerNode
----
...
spec:
driverConfig:
driverType: vSphere
vSphere:
maxAllowedBlockVolumesPerNode: <1>
...
----
<1> Enter the desired value here for the maximum number of volumes per node. The default is 59. The minimum value is 1 and the maximum value is 255.

. Click *Save*.
11 changes: 10 additions & 1 deletion storage/container_storage_interface/persistent-storage-csi-vsphere.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ To create CSI-provisioned persistent volumes (PVs) that mount to vSphere storage

* *vSphere CSI Driver Operator*: The Operator provides a storage class, called `thin-csi`, that you can use to create persistent volumes claims (PVCs). The vSphere CSI Driver Operator supports dynamic volume provisioning by allowing storage volumes to be created on-demand, eliminating the need for cluster administrators to pre-provision storage. You can disable this default storage class if desired (see xref:../../storage/container_storage_interface/persistent-storage-csi-sc-manage.adoc#persistent-storage-csi-sc-manage[Managing the default storage class]).

* *vSphere CSI driver*: The driver enables you to create and mount vSphere PVs. In {product-title} 4.18, the driver version is 3.3.1 The vSphere CSI driver supports all of the file systems supported by the underlying Red Hat Core operating system release, including XFS and Ext4. For more information about supported file systems, see link:https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_file_systems/overview-of-available-file-systems_managing-file-systems[Overview of available file systems].
* *vSphere CSI driver*: The driver enables you to create and mount vSphere PVs. In {product-title} 4.19, the driver version is 3.3.1 The vSphere CSI driver supports all of the file systems supported by the underlying Red Hat Core operating system release, including XFS and Ext4. For more information about supported file systems, see link:https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_file_systems/overview-of-available-file-systems_managing-file-systems[Overview of available file systems].

//Please update driver version as needed with each major OCP release starting with 4.13.

Expand Down Expand Up @@ -109,7 +109,16 @@ include::modules/persistent-storage-csi-vsphere-disable-storage-consequences.ado

include::modules/persistent-storage-csi-vsphere-disable-storage-procedure.adoc[leveloffset=+2]

include::modules/persistent-storage-csi-vsphere-increase-max-vols-per-node-overview.adoc[leveloffset=+1]

:FeatureName: Increasing volumes per node
include::snippets/technology-preview.adoc[leveloffset=+2]

include::modules/persistent-storage-csi-vsphere-increase-max-vols-per-node-procedure.adoc[leveloffset=+2]

== Additional resources
* xref:../../storage/container_storage_interface/persistent-storage-csi.adoc#persistent-storage-csi[Configuring CSI volumes]

* link:https://kb.vmware.com/s/article/1025279[Best practices for using VMware snapshots in the vSphere environment]

* link:https://techdocs.broadcom.com/us/en/vmware-cis/vsphere.html[VMware vCenter documentation]