KEP-3751: Update release signoff Checklist before GA

[sunnylovestiramisu]

• One-line PR description: Update Release Signoff Checklist

• Issue link:

• Other comments: Add dashboard to indicate feature health
• E2E test link: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/testsuites/volume_modify.go

/assign @msau42

[xing-yang]

• Can you change the title "Kubernetes Volume Provisioned IO" to "Kubernetes Modify Volume" or "Kubernetes Volume Attributes Class"? It is not just about "provisioned io" any more.

• "Stress test before GA" is in the Beta section. Do we have stress test?

• This is in the PRR section "TODO Upgrade and rollback will be tested when the feature gate will change to beta." Is this still a TODO?

[xing-yang]

It looks like this PR does not contain this merged change: https://github.com/kubernetes/enhancements/pull/5028/files
Do you need a rebase?

[sftim]

/retitle KEP-3751: Update release signoff Checklist before GA

[sunnylovestiramisu]

/assign johnbelamaric

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sunnylovestiramisu
Once this PR has been reviewed and has the lgtm label, please ask for approval from johnbelamaric. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-storage/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sunnylovestiramisu]

Stress tests still WIP: kubernetes/kubernetes#129918

It is set at a low rate though similar to our other stress tests(10 pods), and creating pod+volume, and then modify the volume.

[johnbelamaric]

Ok, everything looks pretty good. The only thing I'd like to see is some updates to the sections below, for example:

1. SLOs: Did beta confirm the stated SLOs are appropriate?

2. For this one:

   Yes. The VAC protection controller will be expensive because it needs to LIST PVCs/PVs but the call volume
   should be low.

   Can you explain a bit further? I think when you say "volume should be low" what you mean is "volume is user-initiated, not triggered by any upstream controller, and is expected to be low in the typical use case".

3. Was this done? Can you include / point to any results?

   Yes, the feature may impact CreateVolume. We will measure this impact during beta and provide feedback to operators.

4. Was this done? Can you include / point to any results?

   Stress tests will determine increase in resource usage at varying amounts of concurrent volume modifications.

[sunnylovestiramisu]

1. Yes it is still appropriate and it is used by AWS's driver in production workloads right now
2. Your understanding is correct, volume modify is triggered by the user changing a PVC's VAC, and most CO's has a rate limit(you can modify once per 4 hours etc.). So we are not expecting high call rate
3. This one needs an update, because not many customers in AWS creating volume with VAC. But that being said, there is no impact to existing CreateVolume if VAC is not set.
4. Please see the other comment

[sunnylovestiramisu]

Updated these in the actual KEP files as well

[xing-yang]

Do you have a link for this stress test?

[sunnylovestiramisu]

No link, it is their internal tests. I have asked Drew to make a comment here with their numbers, they still need to test CreateVolume with VAC at a much higher rate than the stress tests added in the upstream PR.

[AndrewSirenko]

Yeah I'm not comfortable with putting this existing language about tests done on AWS before promoting to beta into the KEP itself.

AWS EBS did not perform official K8s stress tests when helping promote this KEP from alpha to Beta (because stress tests were required for GA, not beta).

I did perform a spot check of modifying 250 volumes concurrently with EBS CSI Driver before EKS flipped VAC feature gate on by default in EKS 1.31.

[AndrewSirenko]

And ideally these stress tests would be performed with CSI Mock driver, so cloud provider and CSI Driver implementations do not affect test results, right?

[AndrewSirenko]

As discussed offline, I just performed the following quick test:

Patched 250 PVCs with a new volumeAttributesClassName at 5 patches per second. Took 100 seconds total for all modifications to finish, bottle-neck being the slow patch rate due to AWS modification limits

EKS 1.32 (VolumeAttributesClass feature gate & v1beta1 storage API group enabled)
external-resizer:v1.12.0
AWS EBS CSI Driver v1.39.0

Resizer container peaked at 30 milli-vCPU and 34 MiB RAM (Negligible usage)

Here's a gist with commands and manifests: https://gist.github.com/AndrewSirenko/4ab8ec5a1eee51674b5d3ac18835f4e9

[xing-yang]

Yes, there's a WIP stress test mentioned here: #5024 (comment)

[gnufied]

Has anyone used/tested quota features of this KEP before we move the whole KEP to GA? This seems like a problem.

[sunnylovestiramisu]

The latest quota change is this: kubernetes/kubernetes#124360

But it did not merged in yet.

[gnufied]

sure. But it seems weird to merge that code directly with GA feature-gate when it was never tested/merged before.

[johnbelamaric]

Ok, I see the stress test is in progress, I would expect that do be done as part of this GA. The other PRR answers seem good. Once there is SIG approval, I can add the approval.