node-role labeling fails

[hryamzik]

I'm trying to apply node labels using the following configuration:

  - name: test
    instanceType: m5.large
    minSize: 1
    maxSize: 2
    desiredCapacity: 2
    labels:
      role: test
      node-role.kubernetes.io/test: ok

According to #580 and #582 using node-role subdomain should the way to apply labels, however I still get a domain error:

AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_FAILED – "Label cannot contain reserved prefixes [kubernetes.io/, k8s.io/, eks.amazonaws.com/] (Service: AmazonEKS; Status Code: 400; Error Code: InvalidParameterException; Request ID: *** Proxy: null)"

Here's eksctl info:

eksctl version: 0.57.0
kubectl version: v1.21.2
OS: darwin

[Callisto13]

Thanks for asking @hryamzik .

I cannot find that code in eksctl anymore, so it looks like it was removed. I will look into why, and if it happened by accident I will put it back 👍

[AnthonyPoschen]

same issue also exists with

eksctl version: 0.56.0
kubectl version: v1.21.2
OS: darwin

[AnthonyPoschen]

probably has something to do with
https://github.com/weaveworks/eksctl/blob/69c1a78f12017f47bcb247f388e9f9442f7da11c/pkg/apis/eksctl.io/v1alpha5/validation.go#L477

eventually drilling down allowing
https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/staging/src/k8s.io/api/core/v1/well_known_labels.go#L51

not sure how this now allows setting the role.

[Callisto13]

I was not able to dig into this further today, and I am away from now until the week after next. Free for anyone in the team or community to pick up 👍

[gohmc]

While eksctl allows node-role.kubernetes.io but the CF EKS module AWS::EKS::Nodegroup/ManagedNodeGroup rejected it. @TBBle has well explained the issue here.

[TBBle]

Indeed, this is the same core issue as #2363, but using Managed Node Groups, so the error is different.

Hence, the solution is blocked on some development work elsewhere; Managed Node Groups might be solved via the same thing that solves #2363, e.g., kubernetes/cloud-provider-aws#110, but it's also possible that AWS could implement a Managed Node Groups-specific solution which would resolve this problem, but not help #2363.

Interestingly, there's no feature request ticket open on https://github.com/aws/containers-roadmap/ for this. The ticket I linked to before was actually for setting the node-role label to the node group name, rather than the label specified in the config, which is a more-specific use-case than what we have here.

Edit: I opened a feature request upstream for EKS Managed Node Groups to support this: aws/containers-roadmap#1451

[cPu1]

probably has something to do with
https://github.com/weaveworks/eksctl/blob/69c1a78f12017f47bcb247f388e9f9442f7da11c/pkg/apis/eksctl.io/v1alpha5/validation.go#L477

eventually drilling down allowing
https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/staging/src/k8s.io/api/core/v1/well_known_labels.go#L51

not sure how this now allows setting the role.

As @gohmc and @TBBle have pointed out, the error is from the Managed Nodegroups API itself and not from a validation performed in eksctl. eksctl is blocked on the upstream API adding support for reserved prefixes in labels.

[Callisto13]

dupe: #1373