Question regarding cross-compilation

[ghostbuster91]

I read that cross-compilation with nix is possible (though it might not always work).
I wanted to try it in this repository but whatever I do I am always getting an error that my nix-daemon cannot build aarch64 which means that I failed do configure cross-compilation.

I wonder if you tried it?

I also came across your post in bananapi forum where you said:

Nix doesn’t do cross-compilation, so any aarch64 system with enough memory for the nix command to and still have enough memory for the compiler to operate. 1GB might work, but you probably are better off with 2GB or more.

Which confuses me quite a bit. Could you shed some light on that?

[nakato]

I read that cross-compilation with nix is possible (though it might not always work).

Nix doesn’t do cross-compilation,

I was misinformed at the time, and the statement is incorrect, however it is simultaneously mostly accurate.

This discourse post sums up the current state well.

There are three options:

• use a remote aarch64-linux builder
• build with qemu-aarch64 (boot.binfmt.emulatedSystems = [ "aarch64-linux" ], this will be fairly slow, but almost everything will be substituted, so unless you have a lot of custom packages, this still works fine )
• use a cross toolchain (nixpkgs.buildPlatform.system = "x86_64-linux"; nixpkgs.hostPlatform.system = "aarch64-linux";, builds are about as fast as native builds, but you won’t have any cache hits and a lot of stuff doesn’t cross-compile)

https://discourse.nixos.org/t/build-root-filesystem-contents-for-different-architecture/31178/2

Though, when I try to actually cross compile, setting buildPlatform and targetPlatform I get infinite recursion, but that's on my personal repo, which has a lot more going on. Also, if that's going to compile everything without cache, then I expect the cross-compiled packages will not be used when performing native nix commands.

"fairly slow" is probably an understatement for binfmt and compiling the kernel, that would be more akin to a pitch drop experiment.

so any aarch64 system with enough memory for the nix command to and still have enough memory for the compiler to operate. 1GB might work, but you probably are better off with 2GB or more.

The nix command needs around 1GB to evaluate nixpkgs in my experiance, so you need that much memory to not OOM, or set up SWAP, which will be slow.

I deploy from a OrangePi 5B with nixos-rebuild switch --flake '~/git/nixos-flakes#bpir3' --target-host root@bpir3.local, because that'll compile the kernel fairly quickly; its CPU is substantially more capable than the Raspberry Pi 4, and it has enough memory that with zram, the tmpfs on /tmp never hits SWAP on disk, and the eMMC storage massively improves the speed of handling coping to/from the nix store. Last time I timed the kernel compilation, a RPi4 was took about 4x as long as the OPi5b to complete; the BPiR3 took about 14x as long.