Maintenance Jobs Do Not Inherit Pod Labels

[coldnfire]

What steps did you take and what happened:

1. Deployed Velero v1.16.1 using Helm chart 9.1.3
2. Configured the values.yaml file to include the label trust-manager.io/mount-bundle: "true" at the top-level podLabels section. This ensures the label is applied to the main Velero pod.
3. Verified that the main Velero pod indeed has the trust-manager.io/mount-bundle: "true" label.
4. Observed that the resulting maintenance job pod does not inherit the trust-manager.io/mount-bundle: "true" label from the Velero deployment, contrary to documentation.
5. Verified that the node-agent also have correctly the label

What did you expect to happen:

According to Velero documentation, maintenance jobs should inherit labels from the Velero deployment. I expected the maintenance job pod to have the trust-manager.io/mount-bundle: "true" label.

The following information will help us better understand what's going on:

Anything else you would like to add:

This issue prevents proper integration with trust-manager. Trust-manager relies on the trust-manager.io/mount-bundle label to inject certificates into the maintenance job pods. The absence of this label causes the maintenance jobs to fail with the following error: tls: failed to verify certificate: x509: certificate signed by unknown authority. This is a critical issue that blocks proper functioning of Velero in environments where trust-manager is used for certificate management.

Environment:

• Velero version : 1.16.1
• Velero features (use velero client config get features): <NOT SET>
• Kubernetes version : v1.30.8
• Kubernetes installer & version: Kubespray 2.27.0
• Cloud provider or hardware configuration: Vsphere 8.0.3
• OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[blackpiglet]

Thanks for reporting this issue.
PR #8930 modified documents about this topic.

Please note that the maintenance job's labels and annotations are not inherited from the Velero deployment anymore.
Instead, Velero keeps allowlists for them.
If you need to add new ones, please submit PRs.

https://github.com/vmware-tanzu/velero/blob/v1.16.1/pkg/util/third_party.go

[coldnfire]

Thank you for your response.

I'm wondering if adding labels and annotations statically is really the most efficient approach. Does this mean that every Velero user who needs custom labels would need to open a pull request?

Perhaps it would be worth considering either reverting to the previous inheritance method or implementing a way to configure these labels through the values.yaml file in your Helm chart.

Thanks again for your help @blackpiglet

[blackpiglet]

Yes, the current approach would be creating PRs to address the user needs.
I cannot recall the exact reason for this change, but it seems that inheriting the Velero deployment's labels and annotations also caused some weird errors.
I agree that some provisioning methods would be more user-friendly.
We didn't expect there would be such a need from users.

I think we can go with the current way, and open this issue to wait for the community's response on this topic.

[coldnfire]

Hello @blackpiglet ,

Thanks for your answer ;)

Have a nice day !

[msfrucht]

@blackpiglet Another unusual issue along similar lines was proxy support. I don't know the software's name but the user was requesting how to add Pod labels to any Pod that communicated off cluster. It used Pod labels to determine which Pods to inject the proxy settings.

[blackpiglet]

@msfrucht
Would you happen to know the label's name?

[kaovilai]

@msfrucht The proxy solution you're referring to is likely one of several Kubernetes-native approaches for corporate proxy configurations. Based on common patterns, it could be:

1. Service Mesh sidecars (Istio, Linkerd) that use labels like:

   • sidecar.istio.io/inject: "true"
   • linkerd.io/inject: enabled

2. Corporate proxy injection tools that use custom labels to inject proxy environment variables (HTTP_PROXY, HTTPS_PROXY, NO_PROXY) into pods, often implemented via:

   • Mutating admission webhooks
   • Operators that watch for specific labels

3. Trust/certificate management solutions (like the trust-manager mentioned in this issue) that inject CA bundles based on labels

The common pattern is that these tools use pod labels/annotations to determine which pods need proxy configuration or sidecar injection. Without the ability to inherit or configure these labels on maintenance jobs, users in corporate environments face challenges like:

• TLS certificate validation failures (as @coldnfire experienced)
• Inability to reach external resources through corporate proxies
• Failed backup/restore operations due to network restrictions

Perhaps a middle-ground solution could be allowing users to specify additional labels/annotations for maintenance jobs through Velero's configuration, similar to how other Kubernetes controllers handle pod template specs. This would provide flexibility without the risks of inheriting all labels indiscriminately.

[msfrucht]

That would work. I'm very flexible on whatever solution anyone proposes. We just need a way to make sure to set DataUpload and DataDownload associated Pods labels and annotation. Unlike the Deployment and Daemonset these cannot be edit without our own controller to watch and edit.

[janfuhrer]

For us, in a default-deny environment, it is essential to set custom labels on every pod.

We use Cilium ClusterwideNetworkPolicies that match on labels such as example.com/allow-s3-access: "true" to grant S3 access. Without the ability to label Velero’s maintenance jobs, these pods cannot match the policy and are blocked from S3.

There are many potential use cases for custom labels, essentially any scenario where an operator or controller needs to select pods in order to perform its function (e.g. trust-manager, networkpolicies, kyverno, ...).

This is a hard requirement in our setup, and we hope defining custom labels for maintenance jobs will be supported in the future.

[blackpiglet]

#9201 (comment)
@janfuhrer
Does this proposal resolve your issue?