Allow providing Ephemeral PVCs on Worker to avoid space limitation of the node disk

[max-r-clever]

Feature description

Hi !

Description

Currently, the worker deployment use an emptyDir volume, which stores temporary data directly on the node's filesystem. This approach can lead to issues with disk space on the node getting filled up by a task and the worker getting killed.

Requested Feature/Enhancement

The ability to set a PVC for the temporary data directory of the worker deployment.

[hebestreit]

We just faced the issue during the startup of the worker (incl. dind) because it got scheduled on a node with less space available.

I thought about setting requests and limits for the ephemeral storage of the worker deployments. Are there any suggestions how to size it properly and to what extend is the storage used at all if kestra.storage.type is set to S3?

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#setting-requests-and-limits-for-local-ephemeral-storage

deployments:
  worker:
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"