Pod in crashloopback off, panic after Checking for IPAM connectivity... instances m8i-flex, m8i, i8g, i8ge

[wsilva]

What happened:

Nodes not become ready.

The aws-node is crashing.

Ipam log:

...
│ aws-node time="2025-09-04T16:13:07Z" level=info msg="Checking for IPAM connectivity... "
│ aws-node panic: runtime error: index out of range [0] with length 0
│ aws-node
│ aws-node goroutine 1 [running]: 
│ aws-node github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd. (*IPAMContext).markUnmanagedNetworkCards(0xc0003794a0, {0xc000579c30, 0x1, 0xc000575a80?}, {0xc000f4f710, 0x1, 0xc000e21560?})

I have tried the instructions but the metrics port does not work, it looks like that the script expect docker to be running but we are using containerd.

Script output:

$ /opt/cni/bin/aws-cni-support.sh

        This is version 0.7.8. New versions can be found at https://github.com/awslabs/amazon-eks-ami/blob/main/log-collector-script/

Trying to collect common operating system logs...
Trying to collect kernel logs...
Trying to collect modinfo... Trying to collect mount points and volume information...
Trying to collect SELinux status...
Trying to collect iptables information... Trying to collect ipvs information...
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port                 CPS    InPPS   OutPPS    InBPS   OutBPS
  -> RemoteAddress:Port

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
ipset not installed

Trying to collect installed packages...
Trying to collect active system services...
Trying to Collect Containerd daemon information...
Trying to Collect Containerd running information...
Trying to Collect Docker daemon information...

        Warning: The Docker daemon is not running.

Trying to collect kubelet information...
Trying to collect nodeadm information...
Trying to collect L-IPAMD introspection information... Trying to collect L-IPAMD prometheus metrics... Trying to collect L-IPAMD checkpoint...
Trying to collect Multus logs if they exist...
Trying to collect sysctls information...
Trying to collect networking infomation... conntrack v1.4.4 (conntrack-tools): 79 flow entries have been shown.
conntrack v1.4.4 (conntrack-tools): 0 flow entries have been shown.

Trying to collect CNI configuration information... cp: cannot stat '/etc/cni/net.d/*': No such file or directory

Trying to collect CNI Configuration Variables from Docker...

        Warning: The Docker daemon is not running.
Trying to collect CNI Configuration Variables from Containerd...
Trying to collect network policy ebpf loaded data...
Trying to collect Docker daemon logs...
Trying to Collect sandbox-image daemon information...
Trying to Collect CPU Throttled Process Information...
Trying to Collect IO Throttled Process Information...
Trying to Collect reboot history...
Trying to Collect Nvidia Bug report... No Nvidia drivers found, nothing to do.

Trying to archive gathered information...

        Done... your bundled logs are located in /var/log/eks_i-05b631f5d1f5c9413_2025-09-04_1557-UTC_0.7.8.tar.gz

****************************************************************************************
* WARNING: The log bundle collected by this script may contain sensitive information.  *
*                                                                                      *
* Please review the contents of the log bundle carefully and redact or obfuscate       *
* any information you do not wish to be accessible before sharing it with others.      *
****************************************************************************************

I have already sent the log file via e-mail referencing this issue.

Environment:

• Kubernetes version (use kubectl version): v1.32.6-eks-931bdca

• CNI Version: v1.20.0

• OS (e.g: cat /etc/os-release):

  NAME="Amazon Linux"
  VERSION="2"
  ID="amzn"
  ID_LIKE="centos rhel fedora"
  VERSION_ID="2"
  PRETTY_NAME="Amazon Linux 2"
  ANSI_COLOR="0;33"
  CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
  HOME_URL="https://amazonlinux.com/"
  SUPPORT_END="2026-06-30"

• Kernel (e.g. uname -a): Linux ip-10-72-244-130.chapter-ped-industrial-shr 5.10.240-238.959.amzn2.x86_64 #1 SMP Thu Aug 7 19:21:39 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

• core-dns: v1.11.4-eksbuild.14

• kube-proxy: v1.32.6-minimal-eksbuild.2

[jaydeokar]

Please update to v1.20.1 for getting this fix

[wsilva]

Thanks for update @jaydeokar

I think that the official doc need to be updated. It recommends v1.20.0-eksbuild.1

[wsilva]

Upgraded aws-node daemonset to 1.20.1 and it's working like a charm.

Thanks @jaydeokar
🙇

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.