EKS Optimized AMI: race condition during early pod startup causes hostPort iptables failures (-m comment)

[deviprasad303]

Description

We are seeing a race condition on freshly booted EKS worker nodes where pods using hostPort fail during PodSandbox creation due to iptables errors:

iptables v1.8.x (nf_tables): Couldn't load match 'comment'

This happens early in node startup, before the node is fully stabilized.

Environment

Node AMIs: EKS Optimized Amazon Linux 2023
Container runtime: containerd
iptables backend: nft (xtables-nft-multi)
Kernel module involved: xt_comment
Affected workloads: DaemonSets using hostPort (e.g., monitoring agents)

On a fresh node:

1. Node boots
2. containerd starts
3. kubelet starts
4. DaemonSet pod with hostPort schedules immediately
5. kubelet calls CreatePodSandbox
6. CNI runs portmap to configure hostPort rules
7. iptables rule using -m comment fails:
   Couldn't load match 'comment'

CNI ADD fails

1. PodSandbox creation fails with .Error Message:Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox: plugin type="portmap" failed (add): unable to create chain CNI-HOSTPORT-SETMARK: running [/usr/sbin/iptables -t nat -C CNI-HOSTPORT-SETMARK -m comment --comment CNI portfwd masquerade mark -j MARK --set-xmark 0x2000/0x2000 --wait]: exit status 2: iptables v1.8.8 (nf_tables): Couldn't load match comment':No such file or directory`
2. Pod remains stuck in Pending / Init:0/X
3. Later on the same node:
4. xt_comment exists on disk
5. iptables -m comment works when run manually

But the affected pods remain stuck . new pods also has the same behavior

[shvbsle]

Thanks for reporting the issue. A question to triage this quickly: Do you know since which release this started happening? (latest is https://github.com/awslabs/amazon-eks-ami/releases/tag/v20260120). Exact AMI ID would also help