Cross compile to FreeBSD

[Curculigo]

There is only one FreeBSD, solid and consistent, unlike the situation on Linux. FreeBSD derivatives are binary compatible with the vanilla FreeBSD and they use the same libraries, very different from Linux distros. Cross compile to FreeBSD will be very easy. A ready to use sysroot can be easily obtained from the FreeBSD install image.

A FreeBSD sysroot can be easily extracted from the FreeBSD installer image.

Or here: https://download.freebsd.org/releases/amd64/amd64/14.1-RELEASE/

I guess you will need base.txz.

[IsaacShelton]

Windows support is almost finished, and then will come FreeBSD.

From what I understand, the BSDs all use the same ELF binary format and SystemV C calling convention as Linux, so the only real part that's different is they each have different syscalls and C standard library implementations.

The ability to emit ELF object files that use SystemV C calling convention has already been implemented, so I believe all that will be needed is the C standard library to link against.

[thearchivalone]

This isn't perfect but using the FreeBSD Vm runner action, one can create a custom FreeBSD sysroot with build dependencies installed directly into it. Here's what I'm currently working with and so far it's been working well enough: https://github.com/bedwardly-down/thunder-bsd/blob/a6920512a72830f50fb86443a69a255153c1a09d/.github/workflows/main.yml#L780

For my uses, I didn't need executables or most of what else was installed in the base system, so I used find to strip out everything but the required dependencies; your mileage will vary. I also am using caching for it. Getting the dependencies required for building here does require having a FreeBSD local installation and running pkg build-depends <whatever dependency needs to be built from source> and will require manually tweaking. One big build dependency that will bloat the sysroot: if anything needs LLVM, most packages don't support the -lite flavor so it will add 2-3 GB of size to it unless you're willing to manually go in and remove whatever you don't need.

You can build binaries with the runner but expect an extra 15+ minutes every build unless you use heavy caching due to it taking up to 5 minutes every build for the runner to be set up and load and how much extra overhead running a VM on top of a VM would add.

[IsaacShelton]

Yeah FreeBSD support was recently added, so this is something that we will need. Right now CI/CD for automatic testing and nightly builds hasn't been set up yet, but I was thinking about something similar.

https://github.com/vmactions/freebsd-vm seemed promising for CI/CD, although I'm not sure how much it would differ from your approach. Also yes, unfortunately the long build times will be the trade off for the free compute. It would be nice to have standalone testing infrastructure someday, although it's also a lot of work.

What would be the most interesting, is to figure out how to cross-link to FreeBSD without a VM. I think it should be possible (or even trivial) with something like lld and a few .a/.so files from sysroot, although I haven't done the research yet into which specific files are needed.

Also, I'm fairly unfamiliar with the FreeBSD ecosystem, and it sounds like there can be breaking ABI changes between every major FreeBSD version unfortunately (which is very frequent), so that would mean we would need different .a/.so files for every supported major version. Also, there doesn't seem to be a lot of support for legacy FreeBSD binaries, so the benefit of being able to cross-compile to FreeBSD would be pretty limited as I understand it, although it would still be good to have. Correct me if I'm wrong

[thearchivalone]

My approach uses that to build the sysroot. There's overhead when building with that directly.

On the breakage, that's very true. Version 14.1 vs 14.0 are actually two different OS's built on the same kernel branch, if that makes sense. They're not binary backwards compatible

[thearchivalone]

Testing would need the VM, though. There is a hurdle that I have yet to figure out: a test failure in the runner doesn't guarantee a failure in the workflow. It often ignores them and then shows that all tests passed without one looking in the logs.

[IsaacShelton]

Ah okay, that is interesting.

Also that's good to know that even minor versions aren't binary compatible.

For tests failing and the action not failing, would it be possible to indicate by using a file or something? Since it sounds like it mounts the source code tree from the host machine.

It wouldn't be pretty, but something like:

freebsd vm

touch FREEBSD_VM_WAS_SUCCESSFUL.txt   # At end of script

ubuntu-latest

sleep 5 # Just in case
[ ! -f "FREEBSD_VM_WAS_SUCCESSFUL.txt" ] # Make sure the VM was successful

[thearchivalone]

Ah okay, that is interesting.

Also that's good to know that even minor versions aren't binary compatible.

For tests failing and the action not failing, would it be possible to indicate by using a file or something? Since it sounds like it mounts the source code tree from the host machine.

It wouldn't be pretty, but something like:

freebsd vm

touch FREEBSD_VM_WAS_SUCCESSFUL.txt   # At end of script

ubuntu-latest

sleep 5 # Just in case
[ ! -f "FREEBSD_VM_WAS_SUCCESSFUL.txt" ] # Make sure the VM was successful

Yup. That's one solution that could work. Until I discovered that loading a cache had a lookup-only flag, I was doing the same. You can have unlimited caches as long as they don't go over 10GB. Empty "flag files" cache at around 200 bytes each.

Once Version 15.0 releases, FreeBSD is dropping all but 64 bit OS support for that branch on. The end of life 14 release will be the final multi-arch release.

[thearchivalone]

Also, the SSH sync mounts the tree; RSYNC copies from host to VM and then copies back after compilation is done. It has more steps involved but is overall faster, especially on a bigger tree. SSH mounting will add quite a bit of delay. But, on the other hand, if you use RSYNC, any compiled libraries with symlinks or things like ccache will not properly sync back unless you archive them and then decompress back on host (or go directly into the VM runner scripts and manually change the rsync parts of the script).

If you want to see a possible solution, I have a hacked on version here: https://github.com/bedwardly-down/freebsd-vm . It's not perfect and was originally going to be directly integrated into the build system chain to solve a very particular quirk of the runner, but right now, it's just kind of there.

Can this help?

thepowersgang/mrustc#331