Workspace volume gets created even if Profile resourceQuotas block a Notebook from creating a Pod.

[ReggieCarey]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

Was testing Profile based resourceQuotaLimits. So I added the following to my Profile:

  resourceQuotaSpec:
    hard:
      cpu: "8"
      memory: 64Gi
      nvidia.com/gpu: "0"
      persistentvolumeclaims: "9"
      requests.storage: 100Gi

I then purposefully exceeded the total number of CPU's my namespace is allowed.

What did you expect to happen:

I expected to see a notebook spinning for ever and never creating a pod. Further I expected no volumes to be created. After deleting the Notebook that never ran via the GUI, I discovered that a Volume (PVC) had been created. I expected that this volume either not be created or that upon deletion of the notebook, the volume would also be removed.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Ideally, I would expect a webhook to reject the creation of the Notebook CRD or make information available to inform the user about the current resource constraints that block notebook creation. Recognizing that K8s is eventually consistent. I don't think rejecting the Notebook CRD is the right path. However, Volume creation really should occur only when a pod for the notebook is created. It appears that volume creation occurs early and without regard to other resource constraints.

If you can't create a pod due to policy enforcement, don't create a PVC for that pod - or at least remove it if the pod was never created.

Environment:

• Kubeflow version: (based on 1.6.0 of kubeflow/manifests based deployment)
• kfctl version: NOT USING
• Kubernetes platform: KubeADM
• Kubernetes version: (use kubectl version): 1.25.4
• OS (e.g. from /etc/os-release): Ubuntu 20.04

[andreyvelich]

/transfer notebooks

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed if no further activity occurs.
Thank you for your contributions.

Members may comment /lifecycle frozen to prevent this issue from being marked as stale.