When aws-node pod is recreated it will load and pin a new eBPF program with new maps, but the tc filters will sometimes still reference the old program

[bob-bins]

What happened:
When aws-node pod is recreated it will load and pin a new eBPF program with new maps, but the traffic control filters associated with the network interfaces will sometimes still reference the old program

How to reproduce it (as minimally and precisely as possible):

UPDATE: these steps actually aren't enough to reproduce the issue in all cases. We have some nodes where rerolling the aws-node pod does not cause it to pin a new eBPF program; however, the fact that this does happen on some nodes does still point to a bug in this repo.

1. Create an EKS cluster
2. Install vpc-cni addon with enableNetworkPolicy: true
3. Create Pod A that's just a hello world server.
4. Create an Ingress Network Policy for Pod A that requires that other pods that want to connect to it have the label test: test.
5. Create Deployment A with the Pod label test: test.
6. Verify that you can curl pod-a-ip:pod-a-port.
   a. Use debug pods (or ssh/exec to the node/Pod A) to view the ebpf program id associated with the host-side of the veth pair with Pod A. Keep a reference of this id.
   b. Also check that this id is the same one that's referenced by the pinned program at /sys/fs/bpf/globals/aws/programs/<replicaset-name>-<namespace>_handle_ingress
7. Delete the aws-node Pod that runs on the same node as Pod A so it gets recreated.
8. Delete the pod in Deployment A so it gets recreated with a new IP
9. Verify that curl pod-a-ip:pod-a-port times out.
   a. Use debug pods to see that the pinned eBPF program at /sys/fs/bpf/globals/aws/programs/<pod-name>-<namespace>_handle_ingress has a different (new) ID while the old program ID still associated with the eBPF program tied to the network interface.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.34.3-eks-ac2d5a0
• CNI Version: v1.21.1-eksbuild.3
• Network Policy Agent Version: v1.3.1-eksbuild.1
• OS (e.g: cat /etc/os-release): Amazon Linux 2023
• Kernel (e.g. uname -a): Linux 6.12.40-64.114.amzn2023.x86_64

[jaydeokar]

Can you collect node logs when you run into this issue and share it with k8s-awscni-triage@amazon.com to debug further into this issue

[kunhwiko]

This looks similar to the issue that we've been running into. We seem to hit this when we make VPC CNI addon upgrades and have enableNetworkPolicy: true. Connections will time out for new clients that have spawned after the CNI upgrade, but existing clients will continue to function. The issue continues to persist over hours or days. Interestingly, when we restart the server pod (i.e. whatever the client is attempting to connect to), it restores the connection for new clients.

For us, we've noticed this in network policy agent upgrades for:

• v1.1.6-eksbuild.2 > v1.2.4-eksbuild.1
• v1.2.4-eksbuild.1 > v1.2.7-eksbuild.1
• v1.2.7-eksbuild.1 > v1.3.1-eksbuild.1

Would that be similar behavior to what you were seeing @bob-bins?

Unfortunately though, we've also been trying to reproduce this with the steps mentioned above, but haven't reliably been able to do so. We're thinking whether a decent amount of network load may actually be required to reproduce this.

[bob-bins]

@kunhwiko yes, it sounds similar. After the aws-node pod is recreated it begins to maintain a new pinned eBPF program, but (sometimes) the TC filters still reference the old program. New client (correctly) get their IPs added to the newly pinned eBPF program, but the TC filter continues to use the old program so only the old IPs continue to be filtered

I didn't explicitly mention this in the issue description, but this means that in some cases Pods may gain network access to Pods that they should not be able to access. If a Pod whose IP is allowed by the old eBPF program is deleted, any new Pod can be created with the same IP as the deleted Pod, causing the new Pod to gain the same network access as the old one.

[kunhwiko]

@bob-bins We faced the same issue again today and have checked the eBPF program referenced by the tc filter and by NPA have different IDs. Only the one maintained by NPA looks to be getting info about newly spawned pods.

We've shipped details along with eks-log-collector outputs via a AWS support case.

[kunhwiko]

Hey folks! It seems a PR with a possible fix was raised but we were wondering if it is getting any traction - aws/aws-ebpf-sdk-go#143.

[viveksb007]

network policy agent will do this re-create of prog and map only in cases where bpf binary is changed.

@kunhwiko the versions you mentioned above includes ebpf binary changes. We investigated internally and found this related to the NPA recovery behavior which causes drift b/w the state which NPA has and actual ebpf dataplane state. We are working on making recovery mechanism of NPA resilient which should solve this issue.