Helm chart installation on Minikube running on Mac: Module kheaders not found in directory /lib/modules/5.15.49-linuxkit

[ADiTuri]

This is my Minikube setup on a mac instance running on intel:

minikube v1.30.1 on Darwin 13.0.1
✨  Automatically selected the docker driver. Other choices: hyperkit, parallels, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.26.3 preload ...
    > preloaded-images-k8s-v18-v1...:  397.02 MiB / 397.02 MiB  100.00% 5.95 Mi
    > gcr.io/k8s-minikube/kicbase...:  373.53 MiB / 373.53 MiB  100.00% 5.60 Mi
🔥  Creating docker container (CPUs=12, Memory=16384MB) ...
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...

I donwloaded the helm chart and installed kepler and while reading the logs I noticed this:

I0425 20:15:48.549265       1 gpu.go:46] Failed to init nvml, err: could not init nvml: error opening libnvidia-ml.so.1: libnvidia-ml.so.1: cann │
│ I0425 20:15:48.744411       1 exporter.go:150] Kepler running on version: 4bbf144                                                                │
│ I0425 20:15:48.744507       1 config.go:172] using gCgroup ID in the BPF program: true                                                           │
│ I0425 20:15:48.744578       1 config.go:174] kernel version:                                                                                     │
│ I0425 20:15:48.744814       1 rapl_msr_util.go:136] input/output error                                                                           │
│ I0425 20:15:48.744930       1 power.go:77] Not able to obtain power, use estimate method                                                         │
│ I0425 20:15:50.286320       1 exporter.go:175] Initializing the GPU collector                                                                    │
│ modprobe: FATAL: Module kheaders not found in directory /lib/modules/5.15.49-linuxkit                                                            │
│ chdir(/lib/modules/5.15.49-linuxkit/build): No such file or directory                                                                            │
│ I0425 20:15:50.293557       1 bcc_attacher.go:70] failed to attach the bpf program: <nil>                                                        │
│ I0425 20:15:50.293597       1 bcc_attacher.go:130] failed to attach perf module with options [-DNUM_CPUS=14 -DSET_GROUP_ID]: failed to attach th │
│ I0425 20:15:50.293611       1 exporter.go:192] failed to start : failed to attach bpf assets: failed to attach the bpf program: <nil>            │
│ I0425 20:15:50.293743       1 exporter.go:217] Started Kepler in 1.549361852s

When exposing the port 9102 via

kubectl port-forward kepler-lgd6k 9102:9102 -n kepler

This is what I get:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.18.1"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.794584e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.794584e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.446716e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 564
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.678416e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.794584e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.030464e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.833856e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 26465
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.030464e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.86432e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 27029
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 16800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 31200
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 63648
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 65280
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.02198e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 524288
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 524288
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.46322e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 8
# HELP kepler_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which kepler_exporter was built.
# TYPE kepler_exporter_build_info gauge
kepler_exporter_build_info{branch="",goversion="go1.18.1",revision="",version=""} 1
# HELP kepler_node_energy_stat Several labeled node metrics
# TYPE kepler_node_energy_stat counter
kepler_node_energy_stat{cpu_architecture="Coffee Lake",node_block_devices_used="0",node_curr_bytes_read="0",node_curr_bytes_writes="0",node_curr_cache_miss="0",node_curr_container_cpu_usage_seconds_total="0",node_curr_container_memory_working_set_bytes="0",node_curr_cpu_cycles="0",node_curr_cpu_instr="0",node_curr_cpu_time="0",node_curr_energy_in_core_joule="0",node_curr_energy_in_dram_joule="0",node_curr_energy_in_gpu_joule="0",node_curr_energy_in_other_joule="0",node_curr_energy_in_pkg_joule="0",node_curr_energy_in_uncore_joule="0",node_name="minikube"} 0
# HELP kepler_node_nodeInfo Labeled node information
# TYPE kepler_node_nodeInfo counter
kepler_node_nodeInfo{cpu_architecture="Coffee Lake"} 1
# HELP kepler_node_other_host_components_joules_total Aggregated RAPL value in other components (platform - package - dram) in joules
# TYPE kepler_node_other_host_components_joules_total counter
kepler_node_other_host_components_joules_total{instance="minikube",mode="dynamic"} 0
kepler_node_other_host_components_joules_total{instance="minikube",mode="idle"} 0
# HELP kepler_node_platform_joules_total Aggregated RAPL value in platform (entire node) in joules
# TYPE kepler_node_platform_joules_total counter
kepler_node_platform_joules_total{instance="minikube",mode="dynamic",source="acpi"} 0
kepler_node_platform_joules_total{instance="minikube",mode="idle",source="acpi"} 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.59
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 40
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.2214272e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.68245262287e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.3158784e+07
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

Is kepler giving me the right metrics?

[rootfs]

The root cause is this message
modprobe: FATAL: Module kheaders not found in directory /lib/modules/5.15.49-linuxkit │ chdir(/lib/modules/5.15.49-linuxkit/build): No such file or directory

I am not sure why minikube failed on mac while it works on Linux. Is it because the VM image difference?

[AntonioDiTuri]

Minikube should be based on a Docker image right?

[rootfs]

minikube is a VM based kubernetes distribution. There is an (old) instruction of installing kernel headers in order to run ebpf in minikube
https://minikube.sigs.k8s.io/docs/tutorials/ebpf_tools_in_minikube/

[sfontanarrosa]

Hi,
on Mac, Docker use an intermediate VM to run, which has a custom-kernel and the kernel headers are hard to get and/or not available at all. On Linux, Docker use the underlying OS directly.

I was able to by-pass the issue by running Linux on my Mac using Multipass (https://multipass.run) or any other similar (like https://github.com/lima-vm/lima) and running then the k8 cluster (Docker for Linux + k3, Minikube, etc) on top of that. This allowed me to install linux headers since docker was using the Linux virtualized kernel (instead of the docker for Mac VM version).

One clarification: if you have a M1 Mac, the Keeper images are not compiled for arm64 chipsets, so you will need to generated from scratch and install on the local registry for this to work.

Ref:
https://petermalmgren.com/docker-mac-bpf-perf/
https://github.com/sustainable-computing-io/kepler/blob/main/doc/dev/README.md
https://github.com/sustainable-computing-io/kepler/blob/main/build/Dockerfile.builder

[rootfs]

@sfontanarrosa thank you for the info. Can you add this to the kepler-doc?

[sfontanarrosa]

@rootfs for sure. Will submit a PR to the Kepler-doc with some additional how-to for Mac users.

[rootfs]

thank you @sfontanarrosa

[ajgillette]

The root cause is this message modprobe: FATAL: Module kheaders not found in directory /lib/modules/5.15.49-linuxkit │ chdir(/lib/modules/5.15.49-linuxkit/build): No such file or directory

I am not sure why minikube failed on mac while it works on Linux. Is it because the VM image difference?

I'm seeing this on linux as well.