Skip to content

Commit 9c5e55a

Browse files
committed
image/compress-checksum: cap compression threads (default 32)
Threads were set to full nproc, so a 128-core host grabbed ~91 threads for a single large xz job. xz speedup is sublinear past ~16-32 threads and its ratio slightly worsens (more independent blocks), while peak memory grows linearly (~674MB/thread at -9) — ~91 threads reserved ~61GB for marginal wall-time gain. Cap compress_threads to min(nproc, 32). The cap propagates to the xz per-file thread clamp, the xz level-walk memory check, and zstd. Add a COMPRESS_MAX_THREADS override (set >= nproc to disable) and surface the cap in the 'Compression host' diagnostic line. Signed-off-by: Igor Pecovnik <igor@armbian.com>
1 parent 186062c commit 9c5e55a

1 file changed

Lines changed: 9 additions & 2 deletions

File tree

lib/functions/image/compress-checksum.sh

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,16 @@ function output_images_compress_and_checksum() {
2828
# *is* fair-shared among peers, with a 60% safety margin, so the level
2929
# selection below picks the strongest preset that fits without OOMing
3030
# even if every peer also reaches for its top level at the same instant.
31-
declare host_threads active_jobs compress_threads mem_avail_mb mem_budget_mb
31+
declare host_threads active_jobs compress_threads mem_avail_mb mem_budget_mb max_threads
3232
host_threads=$(nproc)
33+
# Cap threads: xz speedup is sublinear and its ratio slightly worsens past
34+
# ~16-32 threads (more independent blocks), while peak memory grows linearly
35+
# (~674MB/thread at -9). On a 128-core box an image would otherwise grab ~91
36+
# threads and reserve ~61GB for marginal wall-time gain. Override per-build
37+
# with COMPRESS_MAX_THREADS (set it >= nproc to disable the cap).
38+
max_threads="${COMPRESS_MAX_THREADS:-32}"
3339
compress_threads=$host_threads
40+
(( compress_threads > max_threads )) && compress_threads=$max_threads
3441
active_jobs=$(pgrep -cx 'xz|zstd|zstdmt' 2>/dev/null || true)
3542
[[ -z "${active_jobs}" ]] && active_jobs=0
3643

@@ -97,7 +104,7 @@ function output_images_compress_and_checksum() {
97104
done
98105

99106
display_alert "Compression host" \
100-
"nproc=${host_threads} loadavg=${loadavg} MemTotal=${mem_total_mb}MB MemAvail=${mem_avail_mb}MB" \
107+
"nproc=${host_threads} (cap=${max_threads}) loadavg=${loadavg} MemTotal=${mem_total_mb}MB MemAvail=${mem_avail_mb}MB" \
101108
"info"
102109
display_alert "Compression resource share" \
103110
"active_xz/zstd=${active_jobs} -> threads=${compress_threads}, budget=${mem_budget_mb}MB; pick xz=-${xz_elastic_level} zstd=-${zstd_elastic_level}" \

0 commit comments

Comments
 (0)