govmomi Client sessions in vSphere are not disconnecting and are left Idle in vmware

[jake15]

Describe the bug
In a Kubernetes Cluster velero has been installed and is connected to a VirtualCenter. When we install this plugin with command "velero plugin add ...velero-plugin-for-vsphere:v1.5.4 -n " the plugin installs successfully and we get the backup-driver and datamgr pods running successfully and we can use the product successfully. However we are seeing our Vsphere connections are not getting closed and are every increasing and we have to manually go in and run some govc code to delete sessions.
https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/v1.5.4/pkg/common/vsphere/virtual_center.go#L108

in the pod/container logs in the datamgr-for-vsphere-plugin we see a new session creation..

datamgr-for-vsphere-plugin-xxxxx ==> datamgr-for-vsphere-plugin time="2025-01-30T15:07:35Z" level=info msg="New session ID for '' = " logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/common/vsphere/virtual_center.go:149"

in vSphere UI we see a new connection for "User Agent" = "cns-dp-useragent" and our "user used to connect" but after a certain amount of time this session just goes Idle and never goes away.

If we delete the pod and the kubernetes daemonset "datamgr-for-vsphere-plugin" respawns, it will create new sessions which can eventually pile up as there is no cleanup happening and you can eventually hit a session limit in vSphere.

To Reproduce
build a kubernetes cluster and install the velero helm chart into it. After this is done install the plugin, you can use this command:
$ velero plugin add ...velero-plugin-for-vsphere:v1.5.4 -n

Once this is up and running and you see all the pods up.. go into the vSphereUI and select "Monitor" + "Sessions" and filter on "User Agent" = "cns-dp-useragent". you will see session Active or Idle here.

Go back into kuberntes cluster and delete a datamgr-for-vsphere-plugin pod and have it respawn. Go back into the vSphereUI and you will see more sessions appear and nothing is getting cleaned up.

Expected behavior
The behavior I would like to see is a better session management used by the govmomi client for this plugin. Right now we have massive session sprawl... I would expect this client being used by this plugin to manage its sessions and if it is not needed to terminate it. I do see a Disconnect function but from the behavior I am seeing it is not being called. https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/v1.5.4/pkg/common/vsphere/virtual_center.go#L242
I would also expect if the pod in kuberenets is terminated (which can happen just by workload movement in k8s cluster) that it should clean up the session.

Troubleshooting Information

[Please refer to the Troubleshooting page and collect the required information]

Screenshots

[If applicable, add screenshots to help explain your problem.]

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue]

[deepakkinni]

@jake15 how often do you see this happening? i understand the issue, but has the Pod restarted that many times that it's flooding all the vc connections or reaching limits?. The fix could be complex, since we are trying to deal with Pod crash, so in theory we may not get an opportunity to trigger the disconnect code.

[jake15]

@deepakkinni We see this happening and leaving a lot of IDLE connections in vmware. I understand it gets complicated because on a pod crash there may be no way to trigger a disconnect...(but I found just deleting a pod or having k8s move its workload around from 1 node to another also leaves sessions) it may be better to go the route of a cleanup process where it looks at idle connections and just cleans them up.. your code already re-triggers a new connection if it needs it... or maybe just close the connection after every action and just create a new one if needed. I work for a company that supports k8s clusters and we rehydrate (add new workers and delete old for security OS patches) every month, so you can image if we don't clean these up every release we keep piling up the connections... Also if a company has a development environment and creates k8s clusters to test and this plugin is part of the build.. it will now spawn idle connection after you just delete the cluster. it seems to me that the connection session logic should get a look at to make sure it cleans up. I had to clean up over 1k sessions with a govc script because no one was looking at this and we hit the max connections limit in vmware.