Reported allocatable volume attachment is erroneous

[Gui13]

/kind bug

What happened?

Reported maximum allocatable volumes depends on the moment where the ebs-csi-node daemonset starts.

On some of our nodes, the reported allocatable volume is extremely low (12, 13, even 2 sometimes), whereas we use instances that should be able to report 25 volumes (we have 1 root disk and 1 eni).

You can see the issue when listing the csinodes objects (here is an example on a 4 nodes cluster):

What you expected to happen?

I expected the driver to expose 25 volumes, just like the other nodes.

How to reproduce it (as minimally and precisely as possible)?

Reboot the ebs-csi-node-xxxx driver on any node where there is already some EBS volumes mounted (from PODs).
The reported allocatable will be lower than what is expected.

This has radical effects:

• impossible to schedule new pods which mount EBS volumes on these nodes
• compounded by the fact that EBS volumes are AZ-dependent, so the scheduler has less and less leeway to schedule these pods
• allocatable limit containing the actual pod volumes makes it coupound even more, since the scheduler counts pod volumes 2 times: in the limit and for its own scheduling constraints

The current workaround that we found is to manually override the node.reservedVolumeAttachments to 1 so that we skip the heuristics about volumes.

I'm actually surprised that nobody has suffered from the problem. We have an extensive use of EBS though, so maybe this is normal..

Anything else we need to know?:

We use the ebs-csi-driver add-on on our EKS cluster with default configuration, provisioned through terraform module.

From a glance in the driver code, it looks like there are two issues:

• when the driver boots, it removes and recreates the ebs driver section in the csinodes with the (erroneous) computation of the allocatable volume attachments
• the heuristics take the number of block devices attached to the node when the driver starts, which includes the EBS volumes attached by the pods. So the volumes are taken "twice": as a reserved volume, but also by the scheduler when deciding if it should schedule new pods (it sees that these pods use a "allocatable" slot).

The faulty code (IMHO) is here:

aws-ebs-csi-driver/pkg/driver/node.go

Line 815 in 79e8b1d

reservedVolumeAttachments = d.metadata.GetNumBlockDeviceMappings() + 1

if reservedVolumeAttachments == -1 {
	// Auto-detect number of reserved volume attachments - plus 1 to account for the root volume
	reservedVolumeAttachments = d.metadata.GetNumBlockDeviceMappings() + 1
}

This counts ALL block devices instead of those that are mounted by default.
The driver can boot at ANY point in the node lifecycle, so it cannot be expected to receive a correct number of devices if started after pods have been scheduled on it.

Environment

• Kubernetes version (use kubectl version): 1.32
• Driver version: 1.53 (but reproduced on 1.52 as well)

[Gui13]

Here is an example on another bigger cluster (~30 nodes)

[ElijahQuinones]

Hi @Gui13 ,

This is a known limitation of our IMDS metadata source. As block device mappings is determined on startup and we have no way of knowing which volumes are CSI managed volumes on startup these are counted twice as you pointed out. The work around for this is exactly as you mentioned setting node.reservedVolumeAttachments to the appropriate value to avoid this issue. We are working on a new metadata source which will mitigate this issue that is currently in Alpha see #2591 .

Also just to note this will only occur when you reboot your node with csi volumes attached to it. Because IMDS only updates attached ebs volumes on instance start.

[Gui13]

Hello @ElijahQuinones , I understand what can lead to that: you mean that on reboot the EC2 instance "remounts" its pod volumes? Shouldn't it mount with only what we asked it to mount (ie: its boot drive)?

There's also a case for the ebs-csi-node pod reboot case, independently from the node reboot itself. This happens, and this is the issue we have on our side.

In this case, the driver seems to be refreshing the csinode allocatable even though the value already exists, whereas the node didn't reboot and has mounted pods with associated volumes. And this number will never get up to 25 unless we completely drain the node.

[ElijahQuinones]

@Gui13

I understand what can lead to that: you mean that on reboot the EC2 instance "remounts" its pod volumes? Shouldn't it mount with only what we asked it to mount (ie: its boot drive)?

No ec2 does not detach volumes on a reboot thus all volumes attached will remain but IMDS will update the block device mappings to reflect the volumes attached to the instance on restart.

There's also a case for the ebs-csi-node pod reboot case, independently from the node reboot itself. This happens, and this is the issue we have on our side.

This should not be possible if the node has not rebooted before the driver has as the block device mappings are static until the instance has been rebooted.

From the public docs

block-device-mapping/ebsN	"The virtual devices associated with any Amazon EBS volumes. Amazon EBS volumes are only available in metadata if they were present at launch time or when the instance was last started. The N indicates the index of the Amazon EBS volume (such as ebs1 or ebs2)."

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-data-categories

[gmauleon]

Hum stumble onto this, I also have a question related to shared EBS volumes when using the VPC CNI addon. Not sure this is taken in consideration in the computation since from my understanding ENI attachments is dynamic depending on the pods present on the node with VPC CNI.

So I think it might be my current problem where the node as the "correct" 25 allocatable but it's less than that in reality after a bunch of pods ends-up on the node.

Dunno if that is taken in consideration in the new Alpha version?

[ConnorJC3]

@gmauleon for ENI related issues, please read this FAQ entry: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md#volume-attachment-capacity-issues

[gmauleon]

@gmauleon for ENI related issues, please read this FAQ entry: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md#volume-attachment-capacity-issues

Got it thanks for both response, since the EKS CSI addon is already using IMDS, I believe we just need to eventually update to 1.34 to solve our problem, it's a fairly recent EKS release so we'll re-test it when we are updated on all our clusters.
Thanks again.