Big spikes on CPU and memory of the agent under high load

[nclaeys]

What happened:

At one of our customers we see huge spikes in memory and CPU of the node policy agent when the cluster is under high load. The customer runs a lot of short-lived jobs. I can reproduce this behavior on our staging cluster by simulating high load with a lot of short jobs.

If I look at the memory metrics of the network policy agent, under normal operation it is around 75Mb, but under load it spikes to 300-700Mb. Attached a graph of the aws-node pod during my load test. The cni never uses more than 50Mb.

Next to the memory also the CPU spikes from 10m to 80-100m:

This is problematic as we initially limitted the memory/cpu of the aws-node but this caused the aws-cni to crashloop and brought the whole cluster down as new pods did not receive an IP address and thus stayed pending for a long time.

In my opinion such a spread is unworkable. What are the recommended request and limit for cpu/memory of the node-policy agent? Do you have any load test results for me to compare our situation to?

Attach logs
I ran the eks-log-collector.sh on the host, so I can send these logs if it helps?

What you expected to happen:

• is this expected behavior?
• I would like to understand why the network-policy-agent is using so much memory/cpu?
• Ideally I want a stable resource consumption from the network-policy-agent.
• I will settle for a recommendation on memory/cpu limit for the component

How to reproduce it (as minimally and precisely as possible):

• my load test has 4 threads, each creating 50 pods that just sleep for 10 seconds.
• for each thread after the 50 pods are finished, it launches again 50 pods.

Anything else we need to know?:
I looked at the similar issue regarding high cpu and memory. We do not have network policy on standalone pods. For us it is used on statefulsets, to make sure our regular workloads cannot reach them, only our custom components running in specific namespaces can.

We noticed the issue after upgrading to Kubernetes v1.32.3 but I do not see the link.
I tried the same test with an older version of the network policy agent, v1.2.0-eksbuild.1 but saw no difference.

Environment:

• Kubernetes version (use kubectl version): v1.32.3-eks-4096722
• CNI Version: v1.19.5-eksbuild.1
• Network Policy Agent Version: v1.2.1-eksbuild.1
• OS (e.g: cat /etc/os-release):

NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.7.20250512"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2029-06-30"

• Kernel (e.g. uname -a):

Linux ip-10-2-158-107.eu-west-1.compute.internal 6.1.134-152.225.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May  7 09:10:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[Pavani-Panakanti]

@nclaeys Can you please share the node log bundle collected to "k8s-awscni-triage@amazon.com" along with the network policies applied to the cluster and cluster arn ? Thanks

[viveksb007]

Hey @nclaeys
few questions related to your reproduction scenario,

• how did you created those short lived pods?
• were they standalone pods or were part of a deployment or job?

Also, can you share the kind of workload you are creating with short lived jobs? we are aware of memory issue due to standalone pods in the system and are tracking the fix here #327

I want to understand how short lived jobs which are not standalone deployment causing the memory spike. What workload you are running should help here.

For CPU spike, we are aware and working on the fixes. Whenever there is high load/churning of pods which are part of NP. There is CPU spikes as NPA does more work during the churn. We are working on optimizing this part to prevent CPU spikes.

[nclaeys]

@Pavani-Panakanti I send the detailed logs to the email address.

@viveksb007 Thanks for reaching out, here are the answers to your questions:

how did you created those short lived pods?

I used our internal container concept to launch these jobs. Based on that, our controller creates a pod from it.
The container concept is similar to the job concept but more tailored for our batch workloads.

were they standalone pods or were part of a deployment or job?

They were not part of a deployment or job but are managed by our Container concept. The network policies are not attached to these standalone pods but are attached to our IDEs, as we do not want that regular jobs can access IDEs. The IDEs are created as statefulsets and an example network policy looks as follows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  creationTimestamp: "2025-05-21T13:01:25Z"
  finalizers:
  - networking.k8s.aws/resources
  generation: 1
  labels:
    app.kubernetes.io/managed-by: conveyor-operator
  name: ide-6c8a7828-aa0a-492e-ab45-6c014683465e
  namespace: dev
  ownerReferences:
  - apiVersion: ide.datafy.cloud/v1
    blockOwnerDeletion: true
    controller: true
    kind: IDE
    name: 6c8a7828-aa0a-492e-ab45-6c014683465e
    uid: ce0ab260-1db9-47dd-a3f9-8450f3a1e26e
  resourceVersion: "855056840"
  uid: 8ab0ed54-d853-428e-9e36-9f82f5ff4f9d
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: datafy
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - port: 8000
      protocol: TCP
    - port: 8001
      protocol: TCP
  podSelector:
    matchLabels:
      ide.datafy.cloud/statefulSet: ide-ce0ab260-1db9-47dd-a3f9-8450f3a1e26e
  policyTypes:
  - Ingress
  - Egress

Also, can you share the kind of workload you are creating with short lived jobs? we are aware of memory issue due to

standalone pods in the system and are tracking the fix here #327

At one of our clients we notice it when running dbt jobs (executing remote queries against a DB or DWH) but I reproduced it on our staging environment by just having pods that sleep for 15 seconds.

For CPU spike, we are aware and working on the fixes. Whenever there is high load/churning of pods which are part of NP. There is CPU spikes as NPA does more work during the churn. We are working on optimizing this part to prevent CPU spikes.

It seems to me indeed more related to the churn, high number of jobs being created and deleted than to the actual job that they are doing. From all our customers. we encounter the most issues with the one that runs a lot of small jobs and do not see big issues at customers that run a smaller amount of really heavy jobs.

From reading #327 I was under the impression that the issue was mainly related to network policies applied to standalone pods. Does the issue also tackle our setup, where the network policies are tied to a statefulset but we create a lot of standalone pods that should comply with the network policy?

[nclaeys]

@viveksb007 @Pavani-Panakanti any update here? Do the logs or the reproducible case help you to detect an issue?

[viveksb007]

@nclaeys apologies for the delay. I looked at the logs and couldn't find the necessary log files. I see you used eks-log-collector.sh to collect the logs.

In the var_log/aws-routed-eni folder, there should be logs corresponding to network-policy-agent like network-policy-agent.log, plugin.log, etc

However the nodelog bundle provided by you doesn't have those log files.

• How are you running this network-policy-agent? via Managed add-on or custom installation?
• Have you changed anything related to logging in network-policy-agent?
• Can you check on the host that those files exist when network-policy-agent is running?

Related to your reproduction of this issue I reproduced it on our staging environment by just having pods that sleep for 15 seconds,
can you share how you created those short lived pods? and were those standalone pods or similar to the internal container concept you are referring?

Looking at the network-policy provided above, I can see it is applied to the pods which have that statefulset label, it also has a ingress rule for namespace datafy, how many pods are in that namespace (datafy) during the spikes?

[nclaeys]

Hi @viveksb007 ,

Thanks for coming back to this. We run the policy agent as managed addon, but the logs are written to stdout instead of to a file. This way all logs are aggregated in cloudwatch. Unfortunately our retention is only 3 weeks and are gone. I will reproduce it again and provide you with the network-policy logs.

I create tasks using Airflow, for every task Airflow will launch a pot and it will sleep for 10 seconds. This is about the simplest setup for me to create. The pods created by Airflow look as follows:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
    dag_id: sample-python-load-1
    karpenter.sh/do-not-disrupt: "true"
    run_id: scheduled__2025-06-28T00:00:00+00:00
    runtime.datafy.cloud/DatafyInstanceType: mx.nano
    task_id: sleeper-load-test-2
    try_number: "1"
  creationTimestamp: "2025-06-30T13:28:18Z"
  labels:
    airflow-worker: "1"
    airflow_version: 2.11.0
    app.kubernetes.io/managed-by: conveyor-operator
    dag_id: sample-python-load-1
    kubernetes_executor: "True"
    launched-by: datafy-airflow-operator
    run_id: scheduled__2025-06-28T0000000000-080eb05cb
    task_id: sleeper-load-test-2
    try_number: "1"
  name: sample-python-load-1-sleeper-load-test-2-248cxemq
  namespace: loadteststg
  resourceVersion: "884431088"
  uid: ce124e70-84a1-43c4-8ee3-c5989568eb9e
spec:
  containers:
  - args:
    - airflow
    - tasks
    - run
    - sample-python-load-1
    - sleeper-load-test-2
    - scheduled__2025-06-28T00:00:00+00:00
    - --local
    - --subdir
    - DAGS_FOLDER/sample-python/load_test.py
    env:
    - name: KUBERNETES_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: AIRFLOW_TASK_TRY_NUMBER
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels['try_number']
    - name: AIRFLOW_TASK_MAPPED_INDEX
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels['map_index']
    - name: CONNECTION_CHECK_MAX_COUNT
      value: "5"
    - name: DAG_SYNCER_ARGS
      value: aws --artifacts-bucket datafy-dp-artifacts-ilwfvh --dag-configmap airflow-loadteststg-config
        --dag-configmap-namespace loadteststg --logs-root-path /opt/airflow/logs --mode
        single
    - name: AWS_SDK_LOAD_CONFIG
      value: "true"
    - name: AWS_DEFAULT_REGION
      value: eu-west-1
    - name: LOG_LEVEL
      value: trace
    - name: AWS_REGION
      value: eu-west-1
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::674971662945:role/conveyor-dag-syncer-datafy-dp-eks-dm-stg
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AIRFLOW_IS_K8S_EXECUTOR_POD
      value: "True"
    envFrom:
    - configMapRef:
        name: airflow-loadteststg-env
    - secretRef:
        name: airflow-loadteststg-env
    image: 674971662945.dkr.ecr.eu-west-1.amazonaws.com/datafy/data-plane/airflow:2-1.18.44-56-g68e4bd5
    imagePullPolicy: IfNotPresent
    name: base
    resources:
      limits:
        cpu: "1"
        memory: 436972Ki
      requests:
        cpu: 105m
        memory: 436972Ki
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 50000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/airflow-config
      name: airflow-config
      readOnly: true
    - mountPath: /tmp
      name: tmp
    - mountPath: /opt/airflow/logs
      name: logs-data
    - mountPath: /opt/airflow/pkgs
      name: package-data
      readOnly: true
    - mountPath: /opt/conveyor/cache
      name: token-cache
    - mountPath: /opt/airflow/xcom
      name: xcom
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-4qsmm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-2-150-13.eu-west-1.compute.internal
  nodeSelector:
    karpenter.sh/capacity-type: spot
    node.kubernetes.io/datafy-instance-class: mx.xlarge
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: conveyor-scheduler
  securityContext:
    fsGroup: 50000
    runAsUser: 50000
  serviceAccount: airflow-loadteststg
  serviceAccountName: airflow-loadteststg
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: datafy-instance-class
    operator: Equal
    value: mx.xlarge
  volumes:
  - configMap:
      defaultMode: 420
      name: airflow-loadteststg-config
    name: airflow-config
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token
  - emptyDir: {}
    name: tmp
  - name: logs-data
    nfs:
      path: /airflow/logs/loadteststg-loadteststg-e3da1fe0-8ac8-4876-81dd-9421d83fe7f7
      server: fs-093e2e2d9fcd9a01d.efs.eu-west-1.amazonaws.com
  - name: package-data
    nfs:
      path: /airflow/pkgs
      server: fs-093e2e2d9fcd9a01d.efs.eu-west-1.amazonaws.com
  - name: token-cache
    nfs:
      path: /airflow/token-cache/loadteststg-loadteststg-e3da1fe0-8ac8-4876-81dd-9421d83fe7f7
      server: fs-093e2e2d9fcd9a01d.efs.eu-west-1.amazonaws.com
  - name: xcom
    nfs:
      path: /airflow/xcom
      server: fs-093e2e2d9fcd9a01d.efs.eu-west-1.amazonaws.com
  - name: kube-api-access-4qsmm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

The datafy namespace never has many pods (5-10 at all times) as this only contains our management pods. The pods are always created in different namespaces. Under load there are between 140-200 pods being launched in my test namespace. Furthermore in kube-system there are around 100 pods (depends largely on the number of nodes being spun up).

I reproduced the situation again on our stg environment. Peak usage of the network policy agent was about 250Mb. The logs for the network policy agent are:

{"level":"info","ts":"2025-06-30T13:49:24.687Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:621","msg":"Successfully attached Ingress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-43-060wq9ft"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.687Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:24.728Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-42-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:24.728Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-42-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:24.728Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"ingress","progFD: ":81}
{"level":"info","ts":"2025-06-30T13:49:24.728Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:669","msg":"Attempting to do an Ingress Attach ","with progFD: ":81}
{"level":"info","ts":"2025-06-30T13:49:24.730Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:621","msg":"Successfully attached Ingress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-42-ogh1da89"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.730Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:24.740Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-43-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:24.741Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-43-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:24.741Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"egress","progFD: ":117}
{"level":"info","ts":"2025-06-30T13:49:24.741Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:703","msg":"Attempting to do an Egress Attach ","with progFD: ":117}
{"level":"info","ts":"2025-06-30T13:49:24.742Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:636","msg":"Successfully attached Egress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-43-060wq9ft"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.742Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:107","msg":"No active policies present for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-43-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.742Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:116","msg":"Updating pod_state map to default_allow for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-43-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.742Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:874","msg":"Pod has an Ingress hook attached. Update the corresponding map","progFD: ":15,"mapName: ":"ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:24.742Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:888","msg":"Pod has an Egress hook attached. Update the corresponding map","progFD: ":117,"mapName: ":"egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:24.843Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-42-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:24.844Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-42-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:24.844Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"egress","progFD: ":63}
{"level":"info","ts":"2025-06-30T13:49:24.844Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:703","msg":"Attempting to do an Egress Attach ","with progFD: ":63}
{"level":"info","ts":"2025-06-30T13:49:24.845Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:636","msg":"Successfully attached Egress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-42-ogh1da89"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.845Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:107","msg":"No active policies present for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-42-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.845Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:116","msg":"Updating pod_state map to default_allow for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-42-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:24.845Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:874","msg":"Pod has an Ingress hook attached. Update the corresponding map","progFD: ":81,"mapName: ":"ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:24.845Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:888","msg":"Pod has an Egress hook attached. Update the corresponding map","progFD: ":63,"mapName: ":"egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:146","msg":"Received a new reconcile request","req":{"name":"notebook-c751a610-98f9-411f-a4da-2f324fc0bebf-znljc","namespace":"dev"}}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:281","msg":"Processing Policy Endpoint  ","Name: ":"notebook-c751a610-98f9-411f-a4da-2f324fc0bebf-znljc","Namespace ":"dev"}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:528","msg":"Parent NP resource:","Name: ":"notebook-c751a610-98f9-411f-a4da-2f324fc0bebf"}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:751","msg":"Found another PE resource for the parent NP","name":"notebook-c751a610-98f9-411f-a4da-2f324fc0bebf-znljc"}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:530","msg":"Total PEs for Parent NP:","Count: ":1}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:551","msg":"Derive PE Object ","Name ":"notebook-c751a610-98f9-411f-a4da-2f324fc0bebf-znljc"}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:561","msg":"Processing PE ","Name ":"notebook-c751a610-98f9-411f-a4da-2f324fc0bebf-znljc"}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:563","msg":"Adding to current targetPods","Total pods: ":0}
{"level":"info","ts":"2025-06-30T13:49:30.347Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:581","msg":"No more target pods so deleting the entry in PE selector map for ","Name ":"notebook-c751a610-98f9-411f-a4da-2f324fc0bebf-znljc"}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:146","msg":"Received a new reconcile request","req":{"name":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe-rnxfp","namespace":"dev"}}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:281","msg":"Processing Policy Endpoint  ","Name: ":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe-rnxfp","Namespace ":"dev"}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:528","msg":"Parent NP resource:","Name: ":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe"}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:751","msg":"Found another PE resource for the parent NP","name":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe-rnxfp"}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:530","msg":"Total PEs for Parent NP:","Count: ":1}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:551","msg":"Derive PE Object ","Name ":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe-rnxfp"}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:561","msg":"Processing PE ","Name ":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe-rnxfp"}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:652","msg":"Current PE Count for Parent NP:","Count: ":1}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:656","msg":"PE for parent NP","name":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe-rnxfp"}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:563","msg":"Adding to current targetPods","Total pods: ":0}
{"level":"info","ts":"2025-06-30T13:49:30.365Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:581","msg":"No more target pods so deleting the entry in PE selector map for ","Name ":"ide-c6fe9c67-d5dc-4829-9b43-64c4a1410dfe-rnxfp"}
{"level":"info","ts":"2025-06-30T13:49:31.161Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:62","msg":"Received Enforce Network Policy Request for Pod","Name":"sample-python-load-4-sleeper-load-test-29-4yzl452n","Namespace":"loadteststg","Mode":"standard"}
{"level":"info","ts":"2025-06-30T13:49:31.161Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:918","msg":"No map instance found"}
{"level":"info","ts":"2025-06-30T13:49:31.161Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:596","msg":"Got the attachProbesLock for","Pod: ":"sample-python-load-4-sleeper-load-test-29-4yzl452n"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-4-sleeper-load-test-29-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:31.161Z","logger":"ebpf-client","caller":"utils/utils.go:178","msg":"host veth interface found","interface name":"eni72135bd63a4"}
{"level":"info","ts":"2025-06-30T13:49:31.161Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:609","msg":"AttacheBPFProbes for","pod":"sample-python-load-4-sleeper-load-test-29-4yzl452n"," in namespace":"loadteststg"," with hostVethName":"eni72135bd63a4"}
{"level":"info","ts":"2025-06-30T13:49:31.161Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:31.226Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-29-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:31.227Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-29-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:31.227Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"ingress","progFD: ":119}
{"level":"info","ts":"2025-06-30T13:49:31.227Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:669","msg":"Attempting to do an Ingress Attach ","with progFD: ":119}
{"level":"info","ts":"2025-06-30T13:49:31.228Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:621","msg":"Successfully attached Ingress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-29-4yzl452n"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:31.228Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:31.290Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-29-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:31.290Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-29-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:31.290Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"egress","progFD: ":122}
{"level":"info","ts":"2025-06-30T13:49:31.290Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:703","msg":"Attempting to do an Egress Attach ","with progFD: ":122}
{"level":"info","ts":"2025-06-30T13:49:31.291Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:636","msg":"Successfully attached Egress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-29-4yzl452n"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:31.291Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:107","msg":"No active policies present for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-29-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:31.291Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:116","msg":"Updating pod_state map to default_allow for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-29-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:31.291Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:874","msg":"Pod has an Ingress hook attached. Update the corresponding map","progFD: ":119,"mapName: ":"ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:31.291Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:888","msg":"Pod has an Egress hook attached. Update the corresponding map","progFD: ":122,"mapName: ":"egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.440Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:144","msg":"Received Delete Network Policy Request for Pod","Name":"sample-python-load-2-sleeper-load-test-32-266wk778","Namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.440Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:151","msg":"Got the deletePodIdentifierLock for","Pod: ":"sample-python-load-2-sleeper-load-test-32-266wk778"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-2-sleeper-load-test-32-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.440Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-32-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.440Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":98}
{"level":"info","ts":"2025-06-30T13:49:32.440Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-32-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.441Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-32-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.441Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":87}
{"level":"info","ts":"2025-06-30T13:49:32.441Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-32-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-32-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:144","msg":"Received Delete Network Policy Request for Pod","Name":"sample-python-load-2-sleeper-load-test-31-redjxwm0","Namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:151","msg":"Got the deletePodIdentifierLock for","Pod: ":"sample-python-load-2-sleeper-load-test-31-redjxwm0"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-2-sleeper-load-test-31-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-31-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":95}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-31-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-31-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":99}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-31-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-31-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:144","msg":"Received Delete Network Policy Request for Pod","Name":"sample-python-load-1-sleeper-load-test-8-cwcemk93","Namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:151","msg":"Got the deletePodIdentifierLock for","Pod: ":"sample-python-load-1-sleeper-load-test-8-cwcemk93"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-1-sleeper-load-test-8-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-8-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":107}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-8-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-8-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":110}
{"level":"info","ts":"2025-06-30T13:49:32.444Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-8-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-8-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:144","msg":"Received Delete Network Policy Request for Pod","Name":"sample-python-load-2-sleeper-load-test-36-fuqwf3lb","Namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:151","msg":"Got the deletePodIdentifierLock for","Pod: ":"sample-python-load-2-sleeper-load-test-36-fuqwf3lb"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-2-sleeper-load-test-36-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-36-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":79}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-36-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-36-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":106}
{"level":"info","ts":"2025-06-30T13:49:32.451Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-36-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-36-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.464Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:144","msg":"Received Delete Network Policy Request for Pod","Name":"sample-python-load-1-sleeper-load-test-31-uuotpbx1","Namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.464Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:151","msg":"Got the deletePodIdentifierLock for","Pod: ":"sample-python-load-1-sleeper-load-test-31-uuotpbx1"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-1-sleeper-load-test-31-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.464Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-31-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.464Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":113}
{"level":"info","ts":"2025-06-30T13:49:32.464Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-31-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.465Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-31-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.465Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":116}
{"level":"info","ts":"2025-06-30T13:49:32.465Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-1-sleeper-load-test-31-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-1-sleeper-load-test-31-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.468Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:144","msg":"Received Delete Network Policy Request for Pod","Name":"sample-python-load-2-sleeper-load-test-35-v7qtsjdz","Namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.469Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:151","msg":"Got the deletePodIdentifierLock for","Pod: ":"sample-python-load-2-sleeper-load-test-35-v7qtsjdz"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-2-sleeper-load-test-35-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:32.469Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-35-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.469Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":92}
{"level":"info","ts":"2025-06-30T13:49:32.469Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-35-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.469Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-35-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:32.469Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":84}
{"level":"info","ts":"2025-06-30T13:49:32.469Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-35-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-2-sleeper-load-test-35-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:39.803Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:62","msg":"Received Enforce Network Policy Request for Pod","Name":"sample-python-load-4-sleeper-load-test-15-ybknso3b","Namespace":"loadteststg","Mode":"standard"}
{"level":"info","ts":"2025-06-30T13:49:39.803Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:918","msg":"No map instance found"}
{"level":"info","ts":"2025-06-30T13:49:39.803Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:596","msg":"Got the attachProbesLock for","Pod: ":"sample-python-load-4-sleeper-load-test-15-ybknso3b"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-4-sleeper-load-test-15-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:39.803Z","logger":"ebpf-client","caller":"utils/utils.go:178","msg":"host veth interface found","interface name":"eni7abad62b26b"}
{"level":"info","ts":"2025-06-30T13:49:39.803Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:609","msg":"AttacheBPFProbes for","pod":"sample-python-load-4-sleeper-load-test-15-ybknso3b"," in namespace":"loadteststg"," with hostVethName":"eni7abad62b26b"}
{"level":"info","ts":"2025-06-30T13:49:39.803Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:39.850Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-15-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:39.850Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-15-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:39.850Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"ingress","progFD: ":79}
{"level":"info","ts":"2025-06-30T13:49:39.850Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:669","msg":"Attempting to do an Ingress Attach ","with progFD: ":79}
{"level":"info","ts":"2025-06-30T13:49:39.851Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:621","msg":"Successfully attached Ingress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-15-ybknso3b"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:39.851Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:39.897Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-15-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:39.897Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-15-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:39.897Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"egress","progFD: ":86}
{"level":"info","ts":"2025-06-30T13:49:39.897Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:703","msg":"Attempting to do an Egress Attach ","with progFD: ":86}
{"level":"info","ts":"2025-06-30T13:49:39.898Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:636","msg":"Successfully attached Egress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-15-ybknso3b"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:39.898Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:107","msg":"No active policies present for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-15-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:39.898Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:116","msg":"Updating pod_state map to default_allow for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-15-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:39.898Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:874","msg":"Pod has an Ingress hook attached. Update the corresponding map","progFD: ":79,"mapName: ":"ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:39.898Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:888","msg":"Pod has an Egress hook attached. Update the corresponding map","progFD: ":86,"mapName: ":"egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:40.032Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:62","msg":"Received Enforce Network Policy Request for Pod","Name":"sample-python-load-4-sleeper-load-test-14-7nawitng","Namespace":"loadteststg","Mode":"standard"}
{"level":"info","ts":"2025-06-30T13:49:40.033Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:918","msg":"No map instance found"}
{"level":"info","ts":"2025-06-30T13:49:40.033Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:596","msg":"Got the attachProbesLock for","Pod: ":"sample-python-load-4-sleeper-load-test-14-7nawitng"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-4-sleeper-load-test-14-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.033Z","logger":"ebpf-client","caller":"utils/utils.go:178","msg":"host veth interface found","interface name":"eni6e34dd0ba8b"}
{"level":"info","ts":"2025-06-30T13:49:40.033Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:609","msg":"AttacheBPFProbes for","pod":"sample-python-load-4-sleeper-load-test-14-7nawitng"," in namespace":"loadteststg"," with hostVethName":"eni6e34dd0ba8b"}
{"level":"info","ts":"2025-06-30T13:49:40.033Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:40.086Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-14-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:40.086Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-14-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:40.086Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"ingress","progFD: ":89}
{"level":"info","ts":"2025-06-30T13:49:40.086Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:669","msg":"Attempting to do an Ingress Attach ","with progFD: ":89}
{"level":"info","ts":"2025-06-30T13:49:40.087Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:621","msg":"Successfully attached Ingress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-14-7nawitng"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.087Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:40.136Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-14-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:40.136Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-14-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:40.136Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"egress","progFD: ":92}
{"level":"info","ts":"2025-06-30T13:49:40.136Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:703","msg":"Attempting to do an Egress Attach ","with progFD: ":92}
{"level":"info","ts":"2025-06-30T13:49:40.181Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:636","msg":"Successfully attached Egress TC probe for","pod: ":"sample-python-load-4-sleeper-load-test-14-7nawitng"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.181Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:107","msg":"No active policies present for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-14-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.181Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:116","msg":"Updating pod_state map to default_allow for ","podIdentifier: ":"sample-python-load-4-sleeper-load-test-14-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.181Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:874","msg":"Pod has an Ingress hook attached. Update the corresponding map","progFD: ":89,"mapName: ":"ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:40.181Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:888","msg":"Pod has an Egress hook attached. Update the corresponding map","progFD: ":92,"mapName: ":"egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:40.184Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:62","msg":"Received Enforce Network Policy Request for Pod","Name":"sample-python-load-2-sleeper-load-test-42-4bn4x9ce","Namespace":"loadteststg","Mode":"standard"}
{"level":"info","ts":"2025-06-30T13:49:40.184Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:918","msg":"No map instance found"}
{"level":"info","ts":"2025-06-30T13:49:40.184Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:596","msg":"Got the attachProbesLock for","Pod: ":"sample-python-load-2-sleeper-load-test-42-4bn4x9ce"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-2-sleeper-load-test-42-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.184Z","logger":"ebpf-client","caller":"utils/utils.go:178","msg":"host veth interface found","interface name":"enif05bc7ec8e3"}
{"level":"info","ts":"2025-06-30T13:49:40.184Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:609","msg":"AttacheBPFProbes for","pod":"sample-python-load-2-sleeper-load-test-42-4bn4x9ce"," in namespace":"loadteststg"," with hostVethName":"enif05bc7ec8e3"}
{"level":"info","ts":"2025-06-30T13:49:40.184Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:40.232Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-42-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:40.232Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-42-loadteststg_handle_ingress"}
{"level":"info","ts":"2025-06-30T13:49:40.232Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"ingress","progFD: ":95}
{"level":"info","ts":"2025-06-30T13:49:40.232Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:669","msg":"Attempting to do an Ingress Attach ","with progFD: ":95}
{"level":"info","ts":"2025-06-30T13:49:40.233Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:621","msg":"Successfully attached Ingress TC probe for","pod: ":"sample-python-load-2-sleeper-load-test-42-4bn4x9ce"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.233Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:785","msg":"Load the eBPF program"}
{"level":"info","ts":"2025-06-30T13:49:40.279Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:797","msg":"Prog Info: ","Pin Path: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-42-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:40.279Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:801","msg":"PinPath for this pod: ","PinPath: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-2-sleeper-load-test-42-loadteststg_handle_egress"}
{"level":"info","ts":"2025-06-30T13:49:40.279Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:804","msg":"Prog Load Succeeded","for ":"egress","progFD: ":98}
{"level":"info","ts":"2025-06-30T13:49:40.279Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:703","msg":"Attempting to do an Egress Attach ","with progFD: ":98}
{"level":"info","ts":"2025-06-30T13:49:40.280Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:636","msg":"Successfully attached Egress TC probe for","pod: ":"sample-python-load-2-sleeper-load-test-42-4bn4x9ce"," in namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.280Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:107","msg":"No active policies present for ","podIdentifier: ":"sample-python-load-2-sleeper-load-test-42-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.280Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:116","msg":"Updating pod_state map to default_allow for ","podIdentifier: ":"sample-python-load-2-sleeper-load-test-42-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:40.280Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:874","msg":"Pod has an Ingress hook attached. Update the corresponding map","progFD: ":95,"mapName: ":"ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:40.280Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:888","msg":"Pod has an Egress hook attached. Update the corresponding map","progFD: ":98,"mapName: ":"egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:144","msg":"Received Delete Network Policy Request for Pod","Name":"sample-python-load-4-sleeper-load-test-39-h95r7f94","Namespace":"loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"rpc-handler","caller":"rpc/rpc_handler.go:151","msg":"Got the deletePodIdentifierLock for","Pod: ":"sample-python-load-4-sleeper-load-test-39-h95r7f94"," Namespace: ":"loadteststg"," PodIdentifier: ":"sample-python-load-4-sleeper-load-test-39-loadteststg"}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-39-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":47}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-39-loadteststg_handle_ingress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_ingress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_ingress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:751","msg":"Deleting: ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-39-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_egress_pod_state_map"}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:762","msg":"Get storedFD ","progFD: ":50}
{"level":"info","ts":"2025-06-30T13:49:41.060Z","logger":"ebpf-client","caller":"ebpf/bpf_client.go:764","msg":"Found the Program and Map to delete - ","Program: ":"/sys/fs/bpf/globals/aws/programs/sample-python-load-4-sleeper-load-test-39-loadteststg_handle_egress","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_egress_map","Map: ":"/sys/fs/bpf/globals/aws/maps/sample-python-load-4-sleeper-load-test-39-loadteststg_egress_pod_state_map"}

Full logs of the agent and the node will be sent by email again.

[runningman84]

I can also confirm that the aws-eks-nodeagent spikes from time to time:

node_namespace_pod_container:container_memory_working_set_bytes{namespace="kube-system",container="aws-eks-nodeagent",node=~"ip-.*"}

It is hard to monitor these things because prometheus only scrapes every 30s. So the real spike is not visible. But you see that the usage goes up a lot and the pod gets restarted (due to oom) and the memory usage is low afterwards again.

The problem is that during this crash/restart period any other created pod on that given node will have a broken network.

Based on the metrics of the author I do not see any viable setting. In order to have a reliable system you would have to configure a request and limit of up to 1gb. I cannot tell my customers to waste resources like that just for these random spikes.

[nclaeys]

@viveksb007 @Pavani-Panakanti any progress on this?

[jaydeokar]

#417
We did make some improvements with CPU usage here. Can you try with the latest version and see if that improves the usage.

We will continue to investigate the Memory usage issue and look for ways to optimize

[yash97]

For leak generated through churn and network policy apply on them, we did repro this using cron job. and fixed memory leak in this pr #520. We have added integration test similar to this scenario. Can you check if latest version have this issue or not? Would like feedback on that, Fix is present in tag 1.3.4.