/usr/sbin/incus-agent should be statically linked

[breml]

bluefin-dx ships with Incus, but by default the incus-agent binary is not fully working, in particular with VMs.

Entering a VM with incus exec gives the following error:

Error: websocket: bad handshake

The reason for this is, that the incus-agent binary, which is shipped with bluefin linux is not statically linked:

❯ file /usr/sbin/incus-agent
/usr/sbin/incus-agent: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=957004a18c107c93
a8e7252d0c563c03d4131721, for GNU/Linux 3.2.0, stripped

My current workaround is to apply a hotfix for /usr/sbin/incus-agent after every update:

ostree admin unlock --hotfix 
wget https://github.com/lxc/incus/releases/download/v6.18.0/bin.linux.incus-agent.x86_64 -O /usr/sbin/incus-agent 
chmod +x /usr/sbin/incus-agent 

But this is not really a sustainable solution.

References:

• https://discuss.linuxcontainers.org/t/vm-agent-issues-please-help-me-understand-how-it-works/19048/3
• Odd incus exec regression with VMs in Debian packaging lxc/incus#2513 (comment)

[breml]

Relevant upstream bugzilla issue: https://bugzilla.redhat.com/show_bug.cgi?id=2389081

[breml]

Additionally, worth noting are the "Packaging recommendations" in https://linuxcontainers.org/incus/docs/main/packaging/#incus-agent-binaries. It would be great, if additionally to the linux version of the incus-agent binary, also the binaries for other architectures / os could be shipped with Bluefin Linux.

[ahmedadan]

Thanks for the thorough issue report and for tracking down the root cause @breml!

This is indeed an upstream Fedora packaging issue. We simply install incus and incus-agent directly from
Fedora's repositories (you can see this in our https://github.com/ublue-os/bluefin/blob/main/build_files/dx/00-dx.sh#L56-L57), so we inherit whatever packaging decisions they've made.

The fix needs to come from Fedora's side, so the bug you linked (https://bugzilla.redhat.com/show_bug.cgi?id=2389081) is the right place to track this.

Regarding multi-architecture binaries: Fedora's incus-agent package currently only provides a single
/usr/bin/incus-agent binary for the host architecture. The upstream Incus project does provide pre-built
static binaries for multiple architectures (linux/aarch64, linux/x86_64, plus macOS and Windows variants), but Fedora's packaging doesn't include these. I see you've already raised this on the bugzilla as well, that's the right place for it since it's also a packaging decision that needs to come from Fedora.

From the bugzilla discussion, it looks like there's already a patch proposed to enable static linking. Once
that lands in Fedora's repos, it should automatically flow through to bluefin-dx on the next image build.

In the meantime, there are two workarounds: the manual hotfix you described, or building a custom
image that downloads the static binaries from the Incus releases during build time - which would also let you include the multi-arch binaries if cross-architecture VM support is something you need.

We'll keep an eye on the upstream bug.

[breml]

@ahmedadan Thanks for your reply. Me personally, I think Bluefin should go the extra mile here instead of just relying on a fix by Fedora, since Incus is named as an important feature of Bluefin-dx (e.g. https://docs.projectbluefin.io/bluefin-dx/). Shipping it in a half-broken way does not make a lot of sense to me. In particular, I think Bluefin could ship all the necessary multi-architecture binaries in a dedicated directory and then set INCUS_AGENT_PATH, as described in https://linuxcontainers.org/incus/docs/main/packaging/#incus-agent-binaries. But this is just my 2cts.

[castrojo]

That's ideal but we'd need help with that, most of the incus-using core maintainers are on chillops so we're shorthanded on this part of the project.

[breml]

@castrojo What kind of help would you need for this?
Statically compiled binaries for all supported architectures are available from the official Incus Github repository (official releases).

(full disclosure: I work on solutions in the Incus ecosystem on my day job)

[castrojo]

This got me motivated to update the project opportunities: #3423

Basically we just need enough people who use Incus on Bluefin to make sure the stuff works over time.

But in your case, tell us what to do to ship the world's best Incus setup and we'll do it, we just need the help. In the past we've depended on third party COPRs, etc and as we moved off of those features like this fell behind. But if we can just get it right from Incus then that'd be way better for us anyway!

[castrojo]

Is any another linux desktop developer experience shipping incus out of the box? We can totally be louder about incus in general!

[ash0x1b]

This has just been fixed by upstream.

incus-6.18-2.fc43

[breml]

Looking at the build log (https://kojipkgs.fedoraproject.org//packages/incus/6.18/2.fc43/data/logs/x86_64/build.log), it might be ok now at least for the linux agent on the same arch as the host. So I guess, this is a move into the right direction.

[breml]

Is any another linux desktop developer experience shipping incus out of the box? We can totally be louder about incus in general!

Not that I am aware of.

Currently the best experience for Incus comes with Debian + the rpm from Zabbly (https://github.com/zabbly/incus). These make sure, that everything is lined up nicely: QEMU version, EFI / EDK2 patches, incus binaries including multi arch incus-agent, Web UI, etc.