exec into system extension for debug purposes

[knfoo]

I am building a Talos extension that runs a frr daemon on the node.
The extension announces the node IP to the network and receives routes from the upstream bgp speakers.
So when I'm provisioning the node, there is not no access to it before the frr daemon is running. This makes debugging very hard.
I can make the extension log to the console, but that is only logging.

It would be very useful to be able to run the extension locally by some means and to be able to exec it to the extension and run the commands as they were in the real extension once installed.

I have tried to export the image and unpack it. Then I can chroot it the WORKDIR and run the script from there. But I guess that lacks some context.

I have read the https://docs.siderolabs.com/talos/v1.11/build-and-extend-talos/custom-images-and-development/developing-talos and related docs to see if I could figure out how to do this.

Any suggestions on how to do this ?

[smira]

If you don't any hacks, you can build self-contained extension and then copy out the contents of extension rootfs using Dockerfile and run it as a docker container locally:

# Dockerfile
FROM scratch AS container
COPY --from=<extension> /usr/local/.... /

When running on Talos, the best way we have right now is to run a privileged Kubernetes pod (e.g. kubectl debug node/....) and nsenter into the extension namespace to debug.

[knfoo]

Yeah, I can do hacks, but generally IMHO when doing hacks like that you also change how it is run, and it may or may not change behavior.

I tried kubectl debug node/... -it --image=ubuntu but I run in to a permission error.

# nsenter --target $(pgrep -f ext-frr) --mount --uts --ipc --net --pid --user --cgroup
nsenter: cannot open /proc/6121/ns/user: Permission denied

[smira]

You should use --profile=sysadmin to get a privileged Kubernetes pod. (also might need -n kube-system with the default Security Policy).

[knfoo]

Thank you, that did the trick.

Now I just have to figure out how to actually use nsenter with my conatiner.

my dockerfile

FROM quay.io/frrouting/frr:8.4.5 AS base

COPY docker-start /usr/lib/frr/docker-start
COPY daemons /etc/frr/daemons
COPY frr.conf.j2 /etc/frr/frr.conf.j2
RUN apk add --no-cache --update-cache tcpdump gettext py3-pip curl lldpd iputils bind-tools busybox-extras mtr lshw jq && \
    pip3 install --upgrade pip setuptools && \
    pip3 install --no-cache-dir j2cli
RUN chmod +x /usr/lib/frr/docker-start

FROM scratch AS frr
COPY --from=base / /rootfs/usr/local/lib/containers/frr/
COPY frr.yaml /rootfs/usr/local/etc/containers/frr.yaml
COPY manifest.yaml /

WORKDIR /rootfs/usr/local/lib/containers/frr

But when I run nsenter
nsenter --target $(pgrep -f ext-frr) --mount --uts --ipc --net --pid --cgroup /usr/local/lib/containers/frr/bin/bash
it fails with:

Error loading shared library libreadline.so.8: No such file or directory (needed by /usr/local/lib/containers/frr/bin/bash)
Error relocating /usr/local/lib/containers/frr/bin/bash: unstifle_history: symbol not found
Error relocating /usr/local/lib/containers/frr/bin/bash: rl_make_bare_keymap: symbol not found
....
....