Issues adding nodes to the cluster.

[MadhvendraDixit]

Description

The issue that I am facing right now is that, even though Karpenter Instance Node role and Karpenter Controller roles have all the permission required for necessary actions. When the number of pods increase, karpenter launches nodes based on the required compute but those nodes do not attach to the cluster and I do not even see any error in the karpenter pod logs.

[jigisha620]

Can you please share all the logs from when this happened as well as your nodepool and nodeclass configuration?

[MadhvendraDixit]

Can you please share all the logs from when this happened as well as your nodepool and nodeclass configuration?

Yes sure,
2025-01-13T08:45:14.800Z INFO controller.provisioner found provisionable pod(s) {"commit": "f013f7b", "pods": 8}
2025-01-13T08:45:14.800Z INFO controller.provisioner computed new machine(s) to fit pod(s) {"commit": "f013f7b", "machines": 3, "pods": 8}
2025-01-13T08:45:14.815Z INFO controller.provisioner created machine {"commit": "f013f7b", "provisioner": "default", "machine": "default-bpqcn", "requests": {"cpu":"1150m","memory":"1Gi","pods":"6"}, "instance-types": "t4g.large, t4g.medium"}
2025-01-13T08:45:14.818Z INFO controller.provisioner created machine {"commit": "f013f7b", "provisioner": "default", "machine": "default-sqssb", "requests": {"cpu":"1650m","memory":"1536Mi","pods":"7"}, "instance-types": "t4g.large, t4g.medium"}
2025-01-13T08:45:14.818Z INFO controller.provisioner created machine {"commit": "f013f7b", "provisioner": "default", "machine": "default-g9b4l", "requests": {"cpu":"1650m","memory":"1536Mi","pods":"7"}, "instance-types": "t4g.large, t4g.medium"}
2025-01-13T08:45:16.055Z DEBUG controller.machine.lifecycle created launch template {"commit": "f013f7b", "machine": "default-bpqcn", "provisioner": "default", "launch-template-name": "karpenter.k8s.aws/16414183443499091970", "id": "lt-0a02dbc53d1b58d5b"}
2025-01-13T08:45:16.200Z DEBUG controller.machine.lifecycle created launch template {"commit": "f013f7b", "machine": "default-bpqcn", "provisioner": "default", "launch-template-name": "karpenter.k8s.aws/7710324587535318196", "id": "lt-0421ee8c2fc2a1094"}
2025-01-13T08:45:17.683Z INFO controller.machine.lifecycle launched machine {"commit": "f013f7b", "machine": "default-g9b4l", "provisioner": "default", "provider-id": "aws:///ap-south-1c/i-09ab3e712bbeb6fa6", "instance-type": "t4g.medium", "zone": "ap-south-1c", "capacity-type": "on-demand", "allocatable": {"cpu":"1930m","ephemeral-storage":"17Gi","memory":"3187Mi","pods":"17"}}
2025-01-13T08:45:17.684Z INFO controller.machine.lifecycle launched machine {"commit": "f013f7b", "machine": "default-bpqcn", "provisioner": "default", "provider-id": "aws:///ap-south-1c/i-0c245ac4ce2c9baa1", "instance-type": "t4g.medium", "zone": "ap-south-1c", "capacity-type": "on-demand", "allocatable": {"cpu":"1930m","ephemeral-storage":"17Gi","memory":"3187Mi","pods":"17"}}
2025-01-13T08:45:17.684Z INFO controller.machine.lifecycle launched machine {"commit": "f013f7b", "machine": "default-sqssb", "provisioner": "default", "provider-id": "aws:///ap-south-1c/i-0e614a7082214f979", "instance-type": "t4g.medium", "zone": "ap-south-1c", "capacity-type": "on-demand", "allocatable": {"cpu":"1930m","ephemeral-storage":"17Gi","memory":"3187Mi","pods":"17"}}
2025-01-13T08:46:41.548Z DEBUG controller.awsnodetemplate discovered subnets {"commit": "f013f7b", "awsnodetemplate": "default", "subnets": ["subnet-07c3433cda79d6457 (ap-south-1a)", "subnet-01da29930d682869e (ap-south-1b)", "subnet-0f24c228d64a92e35 (ap-south-1c)"]}

these are the logs when there are any provisionable pods , and the machines are launched and healthy but are not attached to the EKS cluster

[jigisha620]

What version of Karpenter are you running in your cluster?

[MadhvendraDixit]

What version of Karpenter are you running in your cluster?

Actually I am using same architecture on stage environment and it is working and when I am using it on prod it is not working.
and the version of karpenter I am using is v0.30.0-rc.0

[saurav-agarwalla]

You may need to look at the node logs to see what's happening. Or maybe share the policy attached to the role.

Either way, I also see that you are running a RC version of Karpenter. Is there a reason to not use the latest stable version?

[MadhvendraDixit]

Saurav the same version was working for my stage cluster so I thought of using the same.
After careful analysis I got the issue the reason because of which the instance is not attaching itself to the cluster but I still do not not know how to solve it.

The instance launched by the Karpenter has all the necessary tags just one tag is missing and it is
aws:eks:cluster-name
Even though I have provided all the permissions to the KarpenterInstanceNodeRole including create tags, add tags and assume role.
Can you help me with this thing?