fix(provisioning): register topologyKeys from env rather than all the custom node labels

Signed-off-by: huangfangfeng <[email protected]>

Author: fangfenghuang

deploy/lvm-operator.yaml

@@ -1371,6 +1371,8 @@ spec:
               value: openebs
             - name: METRICS_LISTEN_ADDRESS
               value: :9500
+            - name: ALLOWED_TOPOLOGIES
+              value: "kubernetes.io/hostname,"
           volumeMounts:
             - name: plugin-dir
               mountPath: /plugin

deploy/yamls/lvm-driver.yaml

@@ -1002,6 +1002,8 @@ spec:
               value: openebs
             - name: METRICS_LISTEN_ADDRESS
               value: :9500
+            - name: ALLOWED_TOPOLOGIES
+              value: "kubernetes.io/hostname,"
           volumeMounts:
             - name: plugin-dir
               mountPath: /plugin

docs/faq.md

@@ -1,6 +1,11 @@
 ### 1. How to add custom topology key
 
-To add custom topology key, we can label all the nodes with the required key and value :-
+To add custom topology key:
+* Label the nodes with the required key and value.
+* Set env variables in the LVM driver daemonset yaml(openebs-lvm-node), if already deployed, you can edit the daemonSet directly.
+* "openebs.io/nodename" has been added as default topology key. 
+* Create storageclass with above specific labels keys.
+
 
 ```sh
 $ kubectl label node k8s-node-1 openebs.io/rack=rack1
@@ -10,12 +15,28 @@ $ kubectl get nodes k8s-node-1 --show-labels
 NAME           STATUS   ROLES    AGE   VERSION   LABELS
 k8s-node-1   Ready    worker   16d   v1.17.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-1,kubernetes.io/os=linux,node-role.kubernetes.io/worker=true,openebs.io/rack=rack1
 
+
+$ kubectl get ds -n kube-system openebs-lvm-node -o yaml
+...
+env:
+  - name: OPENEBS_NODE_ID
+    valueFrom:
+      fieldRef:
+        fieldPath: spec.nodeName
+  - name: OPENEBS_CSI_ENDPOINT
+    value: unix:///plugin/csi.sock
+  - name: OPENEBS_NODE_DRIVER
+    value: agent
+  - name: LVM_NAMESPACE
+    value: openebs
+  - name: ALLOWED_TOPOLOGIES
+    value: "openebs.io/rack"
+
 ```
 It is recommended is to label all the nodes with the same key, they can have different values for the given keys, but all keys should be present on all the worker node.
 
-Once we have labeled the node, we can install the lvm driver. The driver will pick the node labels and add that as the supported topology key. If the driver is already installed and you want to add a new topology information, you can label the node with the topology information and then restart the LVM-LocalPV CSI driver daemon sets (openebs-lvm-node) are required so that the driver can pick the labels and add them as supported topology keys. We should restart the pod in kube-system namespace with the name as openebs-lvm-node-[xxxxx] which is the node agent pod for the LVM-LocalPV Driver.
+Once we have labeled the node, we can install the lvm driver. The driver will pick the keys from env "ALLOWED_TOPOLOGIES" and add that as the supported topology key. If the driver is already installed and you want to add a new topology information, you can edit the LVM-LocalPV CSI driver daemon sets (openebs-lvm-node).
 
-Note that restart of LVM LocalPV CSI driver daemon sets are must in case, if we are going to use WaitForFirstConsumer as volumeBindingMode in storage class. In case of immediate volume binding mode, restart of daemon set is not a must requirement, irrespective of sequence of labelling the node either prior to install lvm driver or after install. However it is recommended to restart the daemon set if we are labeling the nodes after the installation.
 
 ```sh
 $ kubectl get pods -n kube-system -l role=openebs-lvm
@@ -49,12 +70,7 @@ spec:
   - name: local.csi.openebs.io
     nodeID: k8s-node-1
     topologyKeys:
-    - beta.kubernetes.io/arch
-    - beta.kubernetes.io/os
-    - kubernetes.io/arch
-    - kubernetes.io/hostname
-    - kubernetes.io/os
-    - node-role.kubernetes.io/worker
+    - openebs.io/nodename
     - openebs.io/rack
 ```
 
@@ -78,4 +94,4 @@ allowedTopologies:
 
 The LVM LocalPV CSI driver will schedule the PV to the nodes where label "openebs.io/rack" is set to "rack1".
 
-Note that if storageclass is using Immediate binding mode and topology key is not mentioned then all the nodes should be labeled using same key, that means, same key should be present on all nodes, nodes can have different values for those keys. If nodes are labeled with different keys i.e. some nodes are having different keys, then LVMPV's default scheduler can not effectively do the volume capacity based scheduling. Here, in this case the CSI provisioner will pick keys from any random node and then prepare the preferred topology list using the nodes which has those keys defined and LVMPV scheduler will schedule the PV among those nodes only.
+Note that if storageclass is using Immediate binding mode and storageclass allowedTopologies is not mentioned then all the nodes should be labeled using "ALLOWED_TOPOLOGIES" keys, that means, "ALLOWED_TOPOLOGIES" keys should be present on all nodes, nodes can have different values for those keys. If some nodes don't have those keys, then LVMPV's default scheduler can not effectively do the volume capacity based scheduling. Here, in this case the CSI provisioner will pick keys from any random node and then prepare the preferred topology list using the nodes which has those keys defined and LVMPV scheduler will schedule the PV among those nodes only.

docs/storageclasses.md

@@ -314,6 +314,57 @@ allowedTopologies:
      - node-2
 ```
 
+At the same time, you must set env variables in the LVM-LocalPV CSI driver daemon sets (openebs-lvm-node) so that it can pick the node label as the supported topology. It add "openebs.io/nodename" as default topology key. If the key doesn't exist in the node labels when the CSI LVM driver register, the key will not add to the topologyKeys. Set more than one keys separated by commas.
+
+```yaml
+env:
+  - name: OPENEBS_NODE_ID
+    valueFrom:
+      fieldRef:
+        fieldPath: spec.nodeName
+  - name: OPENEBS_CSI_ENDPOINT
+    value: unix:///plugin/csi.sock
+  - name: OPENEBS_NODE_DRIVER
+    value: agent
+  - name: LVM_NAMESPACE
+    value: openebs
+  - name: ALLOWED_TOPOLOGIES
+    value: "test1,test2"
+```
+
+We can verify that key has been registered successfully with the LVM LocalPV CSI Driver by checking the CSI node object yaml :-
+
+```yaml
+$ kubectl get csinodes pawan-node-1 -oyaml
+apiVersion: storage.k8s.io/v1
+kind: CSINode
+metadata:
+  creationTimestamp: "2020-04-13T14:49:59Z"
+  name: k8s-node-1
+  ownerReferences:
+  - apiVersion: v1
+    kind: Node
+    name: k8s-node-1
+    uid: fe268f4b-d9a9-490a-a999-8cde20c4dadb
+  resourceVersion: "4586341"
+  selfLink: /apis/storage.k8s.io/v1/csinodes/k8s-node-1
+  uid: 522c2110-9d75-4bca-9879-098eb8b44e5d
+spec:
+  drivers:
+  - name: local.csi.openebs.io
+    nodeID: k8s-node-1
+    topologyKeys:
+    - openebs.io/nodename
+    - test1
+    - test2
+```
+
+If you want to change topology keys, just set new env(ALLOWED_TOPOLOGIES) .Check [faq](./faq.md#1-how-to-add-custom-topology-key) for more details.
+
+```
+$ kubectl edit ds -n kube-system openebs-lvm-node
+```
+
 Here we can have volume group of name “lvmvg” created on the nvme disks and want to use this high performing LVM volume group for the applications that need higher IOPS. We can use the above SorageClass to create the PVC and deploy the application using that.
 
 The LVM-LocalPV driver will create the Volume in the volume group “lvmvg” present on the node with fewer of volumes provisioned among the given node list. In the above StorageClass, if there provisioned volumes on node-1 are less, it will create the volume on node-1 only. Alternatively, we can use `volumeBindingMode: WaitForFirstConsumer` to let the k8s select the node where the volume should be provisioned.
@@ -327,13 +378,7 @@ user@k8s-master:~ $ kubectl label node k8s-node-1 openebs.io/lvmvg=nvme
 node/k8s-node-1 labeled
 ```
 
-Now, restart the LVM-LocalPV Driver (if already deployed, otherwise please ignore) so that it can pick the new node label as the supported topology. Check [faq](./faq.md#1-how-to-add-custom-topology-key) for more details.
-
-```
-$ kubectl delete po -n kube-system -l role=openebs-lvm
-```
-
-Now, we can create the StorageClass like this:
+Add "openebs.io/lvmvg" to the LVM-LocalPV CSI driver daemon sets env(ALLOWED_TOPOLOGIES). Now, we can create the StorageClass like this:
 
 ```yaml
 apiVersion: storage.k8s.io/v1

pkg/driver/agent.go

@@ -19,6 +19,7 @@ package driver
 import (
 	"errors"
 	"net/http"
+	"os"
 	"strings"
 	"sync"
 
@@ -324,11 +325,22 @@ func (ns *node) NodeGetInfo(
 	 * }
 	 */
 
-	// support all the keys that node has
-	topology := node.Labels
-
 	// add driver's topology key
-	topology[lvm.LVMTopologyKey] = ns.driver.config.NodeID
+	topology := map[string]string{
+		lvm.LVMTopologyKey: ns.driver.config.NodeID,
+	}
+
+	// support topologykeys from env ALLOWED_TOPOLOGIES
+	allowedTopologies := os.Getenv("ALLOWED_TOPOLOGIES")
+	allowedKeys := strings.Split(allowedTopologies, ",")
+	for _, key := range allowedKeys {
+		if key != "" {
+			v, ok := node.Labels[key]
+			if ok {
+				topology[key] = v
+			}
+		}
+	}
 
 	return &csi.NodeGetInfoResponse{
 		NodeId: ns.driver.config.NodeID,