CRD CEL validation fails: undefined field 'namespace' on apply

[Shekharrajak]

What happened?

When applying E2E manifests (kubectl apply --server-side -k artifacts/e2e/manifests), three CRDs fail because their CEL validation rule references
t.metadata.namespace, which is not defined in the CEL schema for that path:
• clustertrainingruntimes.trainer.kubeflow.org
• jobsets.jobset.x-k8s.io
• trainingruntimes.trainer.kubeflow.org

Error :

CustomResourceDefinition.apiextensions.k8s.io "clustertrainingruntimes.trainer.kubeflow.org" is invalid: spec.validation.openAPIV3Schema.properties[s
pec].properties[template].properties[spec].properties[volumeClaimPolicies].items.x-kubernetes-validations[0].rule: Invalid value: ... compilation
failed: ERROR: :1:27: undefined field 'namespace'
| self.templates.all(t, !has(t.metadata.namespace) || size(t.metadata.namespace) == 0)
| ..........................^
ERROR: :1:68: undefined field 'namespace'
| self.templates.all(t, !has(t.metadata.namespace) || size(t.metadata.namespace) == 0)
| ...................................................................^

What did you expect to happen?

Rule: self.templates.all(t, !has(t.metadata.namespace) || size(t.metadata.namespace) == 0)
Message: "namespace cannot be set for VolumeClaimPolicies templates"
Repro: Run make test-e2e-setup-cluster (e.g. with Kind v1.30.0). Fails during kubectl apply -k artifacts/e2e/manifests.
Expected: CRDs apply successfully.
Actual: CEL compiler reports undefined field 'namespace' and the CRDs are rejected.
Suggested fix: Adjust or remove this CEL rule so it does not rely on t.metadata.namespace in a schema context where that field is not defined (or use
a CEL expression that is valid for the OpenAPI schema for volumeClaimPolicies items).

Environment

github CI check in SDK kubeflow/sdk#225

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-trainer-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Shekharrajak]

[Sridhar1030]

Hello @Shekharrajak, I’m new to the Kubeflow/Kubernetes codebase, so apologies if I’m missing something.

From my understanding, the CRD fails because the CEL rule

references t.metadata.namespace

, but

metadata.namespace

is not defined in the OpenAPI schema for volumeClaimPolicies.items. Since CEL is schema-aware, this causes the rule to fail compilation during CRD creation.

Would it be correct to fix this by removing or adjusting this validation rule so it doesn’t reference metadata.namespace at this path?

If this is the right direction, I’d be happy to try contributing a fix please let me know if that would be helpful.

[ShaanveerS]

Some findings:
This seems to have been caused by #3144
That commit introduces the CEL rule referencing t.metadata.namespace into the trainer CRDs (confirmed via git log -S / git blame), and JobSet v0.11.0’s CRD also contains the same rule.

Evidence (absent before / present after)

$ git show 09af1a167^:manifests/base/crds/trainer.kubeflow.org_trainingruntimes.yaml | rg "t\.metadata\.namespace" || echo "absent in parent"
absent in parent
$git show 09af1a167:manifests/base/crds/trainer.kubeflow.org_trainingruntimes.yaml  | rg "t\.metadata\.namespace" && echo "present in bad commit"
rule: self.templates.all(t, !has(t.metadata.namespace) || size(t.metadata.namespace) == 0)
present in bad commit

Repro
make test-e2e-setup-cluster K8S_VERSION=1.30.0

fails during kubectl apply --server-side -k artifacts/e2e/manifests with:
clustertrainingruntimes.trainer.kubeflow.org
trainingruntimes.trainer.kubeflow.org
jobsets.jobset.x-k8s.io

all rejected due to CEL compilation error: undefined field 'namespace' for rule:

self.templates.all(t, !has(t.metadata.namespace) || size(t.metadata.namespace) == 0)

Reverting 09af1a1 makes the same repro pass on K8s 1.30.

Proposed fix (keep dependency bump, unblock CRD apply)

The CEL rule is invalid in this schema context because metadata.namespace is not defined for those template items. To keep the bump while unblocking, I suggest removing/patching out that specific x-kubernetes-validations entry:

manifests/base/crds/trainer.kubeflow.org_trainingruntimes.yaml

manifests/base/crds/trainer.kubeflow.org_clustertrainingruntimes.yaml

Happy to open a PR with the patch set if this direction sounds good.

[andreyvelich]

Let's try to fix this here: kubeflow/sdk#253