info-gatherer: fix 'Too many open files' from cpu*4 parallel workers

On a high-core host the info-gatherer runs cpu_count()*4 ProcessPoolExecutor
workers (512 on a 128-core box). Each worker keeps a pipe open in the parent and
forks a piped config-dump subprocess, so the pool exhausts the open-file limit
when it spawns workers (OSError: [Errno 24] Too many open files).

The binding limit is the *container's* RLIMIT_NOFILE, which defaults to ~1024
soft regardless of the host's (often generous) limit — so this bites in Docker
even when a native build is fine. Two complementary fixes:

- docker.sh: pass the HOST's hard nofile into the container via
  --ulimit nofile=<hard>:<hard>, so the container matches the host. The host hard
  limit never exceeds the kernel's fs.nr_open and the container shares that
  kernel, so the value is always a valid --ulimit. Falls back to 1048576 if the
  host reports 'unlimited'.
- armbian_utils.py: before building the pool, raise the soft RLIMIT_NOFILE to the
  hard limit (best-effort) — fixes native/bare-metal runs with a low soft limit
  and is belt-and-suspenders inside the container.

No change to the cpu*4 worker count.

Signed-off-by: Igor Pecovnik <igor@armbian.com>

Author: igorpecovnik

lib/functions/host/docker.sh

@@ -397,13 +397,29 @@ function docker_cli_build_dockerfile() {
 
 function docker_cli_prepare_launch() {
 	display_alert "Preparing" "common Docker arguments" "debug"
+
+	# The container otherwise inherits the Docker daemon's default open-file limit
+	# (classically 1024 soft), independent of the host's own (possibly generous)
+	# limit. That is too low for the parallel info-gatherer (cpu*4 workers, each
+	# holding pipes) and it dies with "OSError: [Errno 24] Too many open files".
+	# Pass the HOST's hard limit through so the container matches the host. The
+	# host hard limit can never exceed the kernel's fs.nr_open, and the container
+	# shares that kernel, so this value is always a valid --ulimit.
+	declare _docker_nofile_hard
+	_docker_nofile_hard="$(ulimit -H -n 2>/dev/null || true)"
+	[[ "${_docker_nofile_hard}" == "unlimited" || -z "${_docker_nofile_hard}" ]] && _docker_nofile_hard=1048576
+
 	declare -g -a DOCKER_ARGS=(
 		"--rm" # side effect - named volumes are considered not attached to anything and are removed on "docker volume prune", since container was removed.
 
 		"--cap-add=SYS_ADMIN"  # add only required capabilities instead
 		"--cap-add=MKNOD"      # (though MKNOD should be already present)
 		"--cap-add=SYS_PTRACE" # CAP_SYS_PTRACE is required for systemd-detect-virt in some cases @TODO: rpardini: so lets eliminate it @TODO: rpardini maybe it's dead already?
 
+		# Match the host's open-file limit (see above) so the parallel info-gatherer
+		# isn't capped by the container's default 1024 and hit Errno 24.
+		"--ulimit" "nofile=${_docker_nofile_hard}:${_docker_nofile_hard}"
+
 		# Pass env var ARMBIAN_RUNNING_IN_CONTAINER to indicate we're running under Docker. This is also set in the Dockerfile; make sure.
 		"--env" "ARMBIAN_RUNNING_IN_CONTAINER=yes"
 

lib/tools/common/armbian_utils.py

@@ -14,6 +14,7 @@
 import logging
 import multiprocessing
 import os
+import resource
 import re
 import shlex
 import subprocess
@@ -542,6 +543,18 @@ def gather_json_output_from_armbian(command: str, targets: list[dict]):
 		total = len(targets)
 		# get the number of processor cores on this machine
 		max_workers = multiprocessing.cpu_count() * 4  # use four times the number of cpu cores, that's the sweet spot
+		# Each pool worker keeps a pipe open in the parent and forks a config-dump
+		# subprocess (more pipes), so a high core count (e.g. 512 workers on a
+		# 128-core host) blows past the default soft open-file limit and the pool
+		# dies spawning workers ("OSError: [Errno 24] Too many open files"). Raise
+		# the soft RLIMIT_NOFILE to the hard limit so the workers have the fds.
+		try:
+			soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
+			if soft < hard:
+				resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard))
+				log.info(f"Raised open-file limit {soft} -> {hard} for {max_workers} workers.")
+		except (ValueError, OSError) as e:
+			log.warning(f"Could not raise the open-file limit for {max_workers} workers: {e}")
 		log.info(f"Using {max_workers} workers for parallel processing.")
 		with concurrent.futures.ProcessPoolExecutor(max_workers=max_workers) as executor:
 			every_future = []