feat(rootfs): native armhf on aarch64 host via runtime-disable of qemu-arm

Continues #9284 (arm64-compat-vdso extension + custom_kernel_make_params hook).
That PR enabled COMPAT_VDSO in arm64 kernels and wired CROSS_COMPILE_COMPAT
through the build; this PR uses the resulting 32-bit-capable kernels to run
32-bit ARM ELF directly instead of through qemu-arm emulation in the
image-build chroot work.

Disable qemu-arm in /proc/sys/fs/binfmt_misc for the duration of the
image-build so 32-bit ARM ELF falls through to kernel binfmt_elf and
runs natively via CONFIG_COMPAT — typically ~12x faster than qemu-arm
emulation on common ARM cores (Cortex-A53/A55/A72/A73/A76). A cleanup
handler re-enables qemu-arm on build exit (success or failure).

Empirical: helios4 BUILD_MINIMAL armhf rootfs cache-miss build on
RK3568 (Cortex-A55) drops from 60:35 to 19:27 — 3.12x speedup; same
build with rootfs cache-hit drops to 6:38 (~5x).

Activation point is delayed to AFTER mmdebstrap (in
create_new_rootfs_cache_via_debootstrap, right past the bash-presence
check) and also at build_rootfs_and_image entry for the rootfs-cache
hit path. Earlier activation breaks mmdebstrap because its cross-arch
machinery requires a working qemu-arm in binfmt_misc and the chroot is
not yet populated for native exec.

Out-of-scope cases (target arch != armhf, host arch != aarch64) return
silently with no log noise. In-scope cases log the capability-check
outcome at info level: native enabled, already-effective no-op, or
32-bit ARM execution unsupported (no COMPAT_VDSO in kernel, Apple
Silicon, ARMv9-only cores) — falling back to qemu-arm-static
emulation. Both docker and native (no-container) builds qualify;
the killswitch is the user-facing opt-out.

Killswitch: NATIVE_ARMHF_ON_ARM64=no disables this path entirely,
before any detection runs — useful on shared build farms or whenever
opting out is desired (synonyms 'never' and 'disabled' are accepted
by the parser).

Sets ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF=yes when active so that
deploy_qemu_binary_to_chroot / undeploy_qemu_binary_from_chroot skip
copying qemu-arm-static into the chroot during image-stage (the kernel
handles the ELF directly).

Caveat: the container shares /proc/sys/fs/binfmt_misc with the host's
init user-ns (Armbian's image-build container has CAP_SYS_ADMIN
regardless of DOCKER_PRIVILEGED), so disabling qemu-arm affects the
host's other concurrent processes for the duration of the build. Same
applies to native (no-container) builds where the host IS the build
host. Acceptable for single-user builders, problematic for shared
build farms where NATIVE_ARMHF_ON_ARM64=no is the right setting.

Assisted-by: Claude:claude-opus-4.7
Signed-off-by: Igor Velkov <325961+iav@users.noreply.github.com>

Author: iav

lib/functions/main/rootfs-image.sh

@@ -13,6 +13,11 @@ function build_rootfs_and_image() {
 	# get a basic rootfs, either from cache or from scratch
 	get_or_create_rootfs_cache_chroot_sdcard # only occurrence of this; has its own logging sections
 
+	# Cache-hit path also benefits — the extracted rootfs has libc/ld-linux, so kernel
+	# binfmt_elf can run 32-bit ARM ELF natively. Idempotent on cache-miss path where
+	# create_new_rootfs_cache_via_debootstrap already activated this.
+	_native_armhf_setup_binfmt_elf || true
+
 	# deploy the qemu binary, no matter where the rootfs came from (built or cached)
 	LOG_SECTION="deploy_qemu_binary_to_chroot_image" do_with_logging deploy_qemu_binary_to_chroot "${SDCARD}" "image" # undeployed at end of this function
 

lib/functions/rootfs/qemu-static.sh

@@ -17,6 +17,13 @@ function deploy_qemu_binary_to_chroot() {
 		return 0
 	fi
 
+	# Native armhf path is active: kernel binfmt_elf executes 32-bit ARM ELF via
+	# CONFIG_COMPAT, no qemu-arm-static needed inside the chroot.
+	if [[ "${ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF:-no}" == "yes" ]]; then
+		display_alert "Native armhf via binfmt_elf" "skipping qemu binary deployment during ${caller}" "info"
+		return 0
+	fi
+
 	declare src_host="/usr/bin/${QEMU_BINARY}"
 	declare dst_target="${chroot_target}/usr/bin/${QEMU_BINARY}"
 	declare dst_target_bkp="${dst_target}.armbian.orig"
@@ -48,8 +55,19 @@ function undeploy_qemu_binary_from_chroot() {
 	declare dst_target="${chroot_target}/usr/bin/${QEMU_BINARY}"
 	declare dst_target_bkp="${dst_target}.armbian.orig"
 
-	# Check the binary we deployed is there. If not, panic, as we've lost control.
+	# Check the binary we deployed is there. Two reasons it might be missing:
+	#   1. ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF was active when the matching deploy
+	#      ran, so nothing was copied — graceful no-op.
+	#   2. Genuine state loss — panic, we lost control.
+	# We must NOT skip the removal solely on the native-armhf flag, because deploy
+	# may have run before that flag was set (rootfs-create deploys at line 134,
+	# native-armhf flips at line 149); skipping the undeploy in that case leaks
+	# the host's qemu-arm-static into the rootfs cache tarball.
 	if [[ ! -f "${dst_target}" ]]; then
+		if [[ "${ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF:-no}" == "yes" ]]; then
+			display_alert "Native armhf via binfmt_elf" "no qemu binary to remove during ${caller}" "debug"
+			return 0
+		fi
 		exit_with_error "Missing qemu binary during undeploy_qemu_binary_from_chroot from ${caller}"
 	fi
 
@@ -97,6 +115,92 @@ function prepare_host_binfmt_qemu() {
 	return 0
 }
 
+# Native armhf on aarch64 host via runtime-disable of qemu-arm in binfmt_misc.
+# With qemu-arm disabled, 32-bit ARM ELF falls through to kernel binfmt_elf, which
+# executes it natively via CONFIG_COMPAT (~12x faster than qemu-arm emulation on common
+# ARM cores). Under docker --privileged, /proc/sys/fs/binfmt_misc is shared with the
+# host (init user-ns), so this WILL affect host's other concurrent processes for the
+# duration of the build — acceptable for single-user builders, problematic for shared
+# build farms. Setting NATIVE_ARMHF_ON_ARM64=no (or never/disabled) disables the entire
+# code path before any detection runs.
+#
+# Returns 0 on success (caller sets ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF=yes); 1 if the
+# preconditions are not met, the user opted out, or disable failed — caller falls back
+# to the qemu-arm path. Cleanup (re-enable qemu-arm) is registered via add_cleanup_handler.
+function _native_armhf_setup_binfmt_elf() {
+	# Killswitch — accept no/never/disabled as synonyms; bail before any detection runs.
+	case "${NATIVE_ARMHF_ON_ARM64:-auto}" in
+		no | never | disabled) return 1 ;;
+	esac
+
+	# Idempotent: callers may invoke this both after debootstrap (in
+	# create_new_rootfs_cache_via_debootstrap) and after rootfs-cache extract (in
+	# build_rootfs_and_image, for cache-hit). Once active, return success.
+	[[ "${ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF:-no}" == "yes" ]] && return 0
+
+	# This optimisation only applies to armhf-target on aarch64-host. Out-of-scope cases
+	# stay completely silent — no alerts, no syscalls past the guards. Both docker and
+	# native builds qualify; the killswitch above is the user-facing opt-out.
+	[[ "${ARCH}" == "armhf" ]] || return 1
+	[[ "$(arch)" == "aarch64" ]] || return 1
+
+	# In-scope from here on: armhf-on-aarch64, killswitch off — log the capability
+	# check result so the user sees whether the acceleration kicked in.
+
+	# Pre-flight: 32-bit ARM execution must actually work on this host. Fails when the
+	# kernel lacks CONFIG_COMPAT/COMPAT_VDSO (extensions/arm64-compat-vdso, see PR #9284),
+	# or when the CPU itself lacks 32-bit user mode (Apple Silicon, ARMv9-only cores).
+	if ! arch-test armhf > /dev/null 2>&1; then
+		display_alert "Native armhf via binfmt_elf" "32-bit ARM execution not supported on this kernel/host; falling back to qemu-arm-static emulation" "info"
+		return 1
+	fi
+
+	# Need a registered, enabled qemu-arm handler to disable; otherwise the kernel will
+	# already fall through to binfmt_elf — mark active and report the no-op outcome.
+	if [[ ! -e /proc/sys/fs/binfmt_misc/qemu-arm ]]; then
+		display_alert "Native armhf via binfmt_elf" "qemu-arm not registered in binfmt_misc; native armhf already in effect" "info"
+		declare -g ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF=yes
+		return 0
+	fi
+
+	if ! head -1 /proc/sys/fs/binfmt_misc/qemu-arm 2> /dev/null | grep -q '^enabled'; then
+		display_alert "Native armhf via binfmt_elf" "qemu-arm already disabled in binfmt_misc; native armhf already in effect" "info"
+		declare -g ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF=yes
+		return 0
+	fi
+
+	# Register cleanup BEFORE the destructive write so an interrupt mid-setup
+	# (SIGINT / set -e) still re-enables qemu-arm on the host's shared binfmt_misc.
+	# State machine: pending (handler must restore) → yes (success) or no (write
+	# failed, nothing to restore).
+	declare -g _native_armhf_disabled_qemu_arm=pending
+	add_cleanup_handler trap_handler_native_armhf_restore_qemu_arm
+
+	# Disable qemu-arm in the (shared) binfmt_misc; the kernel will fall through to
+	# binfmt_elf for 32-bit ARM ELF and execute natively via CONFIG_COMPAT.
+	if ! echo 0 > /proc/sys/fs/binfmt_misc/qemu-arm 2> /dev/null; then
+		declare -g _native_armhf_disabled_qemu_arm=no
+		display_alert "Native armhf via binfmt_elf" "could not disable qemu-arm (no CAP_SYS_ADMIN?); falling back to qemu-arm-static emulation" "wrn"
+		return 1
+	fi
+	declare -g _native_armhf_disabled_qemu_arm=yes
+
+	display_alert "Native armhf via binfmt_elf" "kernel $(uname -r), aarch64 host with COMPAT_VDSO; qemu-arm disabled, kernel binfmt_elf takes over" "info"
+	declare -g ARMBIAN_NATIVE_ARMHF_VIA_BINFMT_ELF=yes
+	return 0
+}
+
+# Cleanup handler: re-enable qemu-arm after build (whether success or failure).
+# Idempotent — writing 1 to an already-enabled entry is a no-op for binfmt_misc.
+function trap_handler_native_armhf_restore_qemu_arm() {
+	# Skip only when we know we never wrote (write itself failed); pending and yes
+	# both require restore.
+	[[ "${_native_armhf_disabled_qemu_arm:-no}" == "no" ]] && return 0
+	[[ -e /proc/sys/fs/binfmt_misc/qemu-arm ]] || return 0
+	echo 1 > /proc/sys/fs/binfmt_misc/qemu-arm 2> /dev/null || true
+	display_alert "Native armhf via binfmt_elf" "qemu-arm re-enabled in binfmt_misc" "info"
+}
+
 # The actual binfmt manipulations when cross-build is confirmed above.
 function prepare_host_binfmt_qemu_cross() {
 	local failed_binfmt_modprobe=0
@@ -171,7 +275,7 @@ function prepare_host_binfmt_qemu_cross_arm64_host_armhf_target() {
 		display_alert "Host can run armhf natively or emulation is correctly setup already" "no need to enable qemu-arm" "debug"
 	else
 		display_alert "arm64 host can't run armhf natively" "importing enabling qemu-arm" "debug"
-		cat <<-BINFMT_ARM_MAGIC >/usr/share/binfmts/qemu-arm
+		cat <<- BINFMT_ARM_MAGIC > /usr/share/binfmts/qemu-arm
 			package qemu-user-static
 			interpreter /usr/bin/qemu-arm-static
 			magic \x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00

lib/functions/rootfs/rootfs-create.sh

@@ -142,6 +142,12 @@ function create_new_rootfs_cache_via_debootstrap() {
 
 	[[ ! -f "${SDCARD}/bin/bash" ]] && exit_with_error "mmdebstrap did not produce /bin/bash"
 
+	# mmdebstrap done, libc/ld-linux are in ${SDCARD}. Disable qemu-arm in binfmt_misc
+	# so subsequent chroot apt-get/dpkg/customize calls fall through to kernel binfmt_elf
+	# and run 32-bit ARM ELF natively via CONFIG_COMPAT. mmdebstrap above used qemu-arm
+	# because its cross-arch path requires that registration to be present.
+	_native_armhf_setup_binfmt_elf || true
+
 	# Done with mmdebstrap. Clean-up its litterbox.
 	display_alert "Cleaning up after mmdebstrap" "mmdebstrap cleanup" "info"
 	run_host_command_logged rm -rf "${SDCARD}/var/cache/apt" "${SDCARD}/var/lib/apt/lists"