No bpf execution result displayed

[no1asian]

I've install kubectl-trace without error in my K8S cluster.

But no result displayed.

$ kubectl trace run ip-10-250-27-39.eu-central-1.compute.internal -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"
trace e894f5e4-fd28-40a6-bce8-ed3d11c082d4 created

Finished without result data. Any other method or path to check the result?

[rainest]

Are you able to see the trace containers/jobs' output? I had a similar issue running under KIND, where I lacked kernel headers, causing the trace pods to end quickly. It looks like the trace plugin isn't quite able to properly deal with situations where its pods disappear, so it just hangs there and shows nothing, or will indicate it can't find any trace if you try trace logs.

The jobs use a short ttlSecondsAfterFinished value, so seeing the affected pods is a bit difficult. The best option I've found is immediately running kubectl edit job immediately after starting trace and deleting that line, but that probably won't work as well on a cluster with other activity.

--fetch-headers should fix this, but it's a bit outdated at the moment: #179

[mYu4N]

run trace:
~]# kubectl trace run cn-beijing.192.168.88.23 -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"
trace b4bf4d65-957d-11ed-8c73-00163e0c327b created

log pod:
~]# kubectl logs kubectl-trace-b4bf4d65-957d-11ed-8c73-00163e0c327b-9w9z6
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
Attaching 336 probes...

login node ,and use kill send sigint :
~]# pgrep bpftrace
252665
~]# pstree -sp 252665
systemd(1)───containerd-shim(252587)───timeout(252641)───trace-runner(252653)───bpftrace(252665)
~]# kill -2 252653

then,view pods log:

~]# kubectl logs kubectl-trace-b4bf4d65-957d-11ed-8c73-00163e0c327b-9w9z6
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
Attaching 336 probes...

first SIGINT received, now if your program had maps and did not free them it should print them out

so,where find my bpftrace result ?

use trace xxx -a, Still can't see the result
]# kubectl trace run cn-beijing.192.168.88.23 -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }" -a

kill -2 timeout pid ,i can use logs pod to view bpftrace result ,and pod status is completed

]# pstree -sp 257811
systemd(1)───containerd-shim(257734)───timeout(257788)───trace-runner(257800)───bpftrace(257811)

kubectl-trace-d411da1d-957e-11ed-a75c-00163e0c327b-btltq 0/1 Completed

]# kubectl logs kubectl-trace-d411da1d-957e-11ed-a75c-00163e0c327b-btltq
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
Attaching 336 probes...

first SIGINT received, now if your program had maps and did not free them it should print them out

@[tracepoint:syscalls:sys_enter_symlink]: 1
@[tracepoint:syscalls:sys_enter_fallocate]: 1
@[tracepoint:syscalls:sys_enter_restart_syscall]: 1
@[tracepoint:syscalls:sys_enter_time]: 1
@[tracepoint:syscalls:sys_enter_add_key]: 1
@[tracepoint:syscalls:sys_enter_sendmmsg]: 1

[LIRUILONGS]

I have the same problem and I want to know why and where the problem is, any help would be greatly appreciated

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl trace run node/vms103.liruilongs.github.io -f opensnoop.bt
trace fc86c785-e513-11ee-8e04-000c290e5d5f created

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl get pods | grep fc86c785-e513-11ee-8e04-000c290e5d5f
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     ContainerCreating   0             8s

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl get pods kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm -w
NAME                                                       READY   STATUS      RESTARTS   AGE
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Completed   0          19s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Completed   0          20s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Completed   0          21s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Terminating   0          26s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Terminating   0          26s
^C┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl logs  kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm
Error from server (NotFound): pods "kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm" not found

if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two ││ times                                                                                                                │
│/bpftrace/include/clang_workarounds.h:14:10: fatal error: 'linux/types.h' file not found                              ││exit status 1

Add --fetch-headers to try

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl trace run vms105.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }" --fetch-headers
trace 3d9981a0-e59b-11ee-a570-000c290e5d5f created
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods
NAME                                                       READY   STATUS     RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running    1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               12s
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods -w
NAME                                                       READY   STATUS     RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running    1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               33s
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               41s
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods -w
NAME                                                       READY   STATUS     RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running    1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               64s
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods
NAME                                                       READY   STATUS    RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running   1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   1/1     Running   0               16m

---

Os

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$hostnamectl
   Static hostname: vms100.liruilongs.github.io
         Icon name: computer-vm
           Chassis: vm
        Machine ID: e93ae3f6cb354f3ba509eeb73568087e
           Boot ID: 51ad5f1933914654affd2dcf9ebca862
    Virtualization: vmware
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 5.4.266-1.el7.elrepo.x86_64
      Architecture: x86-64

script

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$cat opensnoop.bt
#!/usr/bin/bpftrace
/*
 * opensnoop    Trace open() syscalls.
 *              For Linux, uses bpftrace and eBPF.
 *
 * Also a basic example of bpftrace.
 *
 * USAGE: opensnoop.bt
 *
 * This is a bpftrace version of the bcc tool of the same name.
 *
 * Copyright 2018 Netflix, Inc.
 * Licensed under the Apache License, Version 2.0 (the "License")
 *
 * 08-Sep-2018  Brendan Gregg   Created this.
 */

BEGIN
{
        printf("Tracing open syscalls... Hit Ctrl-C to end.\n");
        printf("%-6s %-16s %4s %3s %s\n", "PID", "COMM", "FD", "ERR", "PATH");
}

tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat
{
        @filename[tid] = args->filename;
}

tracepoint:syscalls:sys_exit_open,
tracepoint:syscalls:sys_exit_openat
/@filename[tid]/
{
        $ret = args->ret;
        $fd = $ret >= 0 ? $ret : -1;
        $errno = $ret >= 0 ? 0 : - $ret;

        printf("%-6d %-16s %4d %3d %s\n", pid, comm, $fd, $errno,
            str(@filename[tid]));
        delete(@filename[tid]);
}

END
{
        clear(@filename);
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$