Skip to content

Commit 1b78f82

Browse files
igorpecovnikigorpecovnik
authored
Improve speed and reliability of repo management (#157)
* Implement new parallel repository workflow in GitHub Actions Updated infrastructure-repository-update.yml to use the new 3-step parallel repository architecture that eliminates duplicate work. Workflow changes: - Step 1: main job builds common component with update-main - Step 2: postclean jobs build release-specific components in parallel using -R flag (isolated DBs, no publishing) - Step 3: NEW merge job combines common + release-specific components and publishes complete repositories - Updated cleanup job dependency to wait for merge Benefits: - Common packages added once instead of per-release - True parallelism across multiple releases - Better error isolation between releases - Scalable architecture for adding more releases Signed-off-by: Igor Pecovnik <igor@armbian.com> * Use dev branch * Revert "Use dev branch" This reverts commit 7f06a08. * Change order * Use testing branch * fix: use UPLOAD_PATH env var and add example.deb to empty repos - Replace DOWNLOAD_PATH with UPLOAD_PATH environment variable - Update UPLOAD_PATH to "storage/incoming/external" for better organization - Update all rsync commands to use ${UPLOAD_PATH} instead of hardcoded "storage/" - Add example.deb to each generated subfolder (main, utils, desktop) to ensure empty repos are properly published with complete structure This ensures repository components are always publishable even when no packages are added to them. Signed-off-by: Igor Pecovnik <igor@armbian.com> * Update * Implement skip downloading if file exist on server * feat: add architecture filtering to remote package check Add architecture-aware filtering when checking for existing packages on remote storage to prevent skipping downloads for wrong architectures. - Map Debian arch names to common filename variants (amd64→x86_64,x64, arm64→aarch64, etc.) - Build find command with architecture-specific patterns - Only skip download if package exists for matching architecture Signed-off-by: Igor Pecovnik <igor@armbian.com> * Drop caching * Fix upload logic --------- Signed-off-by: Igor Pecovnik <igor@armbian.com>
1 parent f0bc381 commit 1b78f82

2 files changed

Lines changed: 147 additions & 65 deletions

File tree

‎.github/workflows/infrastructure-download-external.yml‎

Lines changed: 53 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ env:
6565
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
6666
PR_NUMBER: ${{ github.event.number }}
6767
PACKAGES_URL: "https://fi.mirror.armbian.de/apt"
68+
UPLOAD_PATH: "storage/incoming/external"
6869

6970
jobs:
7071

@@ -219,7 +220,7 @@ jobs:
219220
# Note: StrictHostKeychecking=no is used here; consider using proper known_hosts in production
220221
ssh -o StrictHostKeychecking=no -p ${{ secrets.HOST_UPLOAD_PORT }} \
221222
${{ inputs.HOST_USER }}@${{ inputs.HOST_DEPLOY }} \
222-
"rm -rf storage/artifacts/*"
223+
"rm -rf ${UPLOAD_PATH}/artifacts/*"
223224
224225
- name: Checkout Armbian OS repository
225226
uses: actions/checkout@v6
@@ -309,27 +310,26 @@ jobs:
309310
# Force amd64 runner for aptly method due to stability issues on ARM
310311
if [[ "${METHOD:-aptly}" == "aptly" ]]; then
311312
# Always use amd64 runner for aptly, use QEMU for other arches
312-
runner="ubuntu-latest"
313+
runner='["docker", "X64"]'
313314
image_arch="amd64"
314315
else
315316
# Use architecture-specific runners for other methods (gh, direct)
316317
case "${arch}" in
317318
amd64)
318-
runner="ubuntu-latest"
319-
#runner="X64"
319+
runner='["docker", "X64"]'
320320
image_arch="${arch}"
321321
;;
322322
arm64)
323-
runner="ubuntu-24.04-arm"
324-
#runner="docker"
323+
runner='["docker", "ARM64"]'
325324
image_arch="${arch}"
326325
;;
327326
armhf|riscv64)
328-
runner="ubuntu-latest"
327+
runner='["ubuntu-latest"]'
328+
runner='["docker", "X64"]'
329329
image_arch="amd64" # Use amd64 image with QEMU emulation
330330
;;
331331
*)
332-
runner="ubuntu-latest"
332+
runner='["ubuntu-latest"]'
333333
image_arch="amd64"
334334
;;
335335
esac
@@ -347,7 +347,7 @@ jobs:
347347
--arg image_arch "${image_arch}" \
348348
--arg registry "ghcr.io/${{ github.repository_owner }}" \
349349
--argjson needs_qemu "${needs_qemu}" \
350-
'{name: $name, arch: $arch, release: $release, target: $target, method: $method, install: $install, runner: $runner, image_arch: $image_arch, registry: $registry, needs_qemu: $needs_qemu}'
350+
'{name: $name, arch: $arch, release: $release, target: $target, method: $method, install: $install, runner: ($runner | fromjson), image_arch: $image_arch, registry: $registry, needs_qemu: $needs_qemu}'
351351
done
352352
done
353353
done | jq -s '{"include": .}'
@@ -435,15 +435,20 @@ jobs:
435435
436436
# Install appropriate keyring based on container type
437437
if grep -q "debian" /etc/os-release; then
438-
apt-get -y install debian-keyring
438+
apt-get -y install debian-keyring 2>&1 | grep -v "^gpg:" || true
439439
elif grep -q "ubuntu" /etc/os-release; then
440-
apt-get -y install ubuntu-keyring
440+
apt-get -y install ubuntu-keyring 2>&1 | grep -v "^gpg:" || true
441441
fi
442442
443-
# Import additional keys
444-
gpg --no-default-keyring --keyring trustedkeys.gpg \
443+
# Import additional keys with timeout and suppress verbose output
444+
# Use timeout to prevent hanging on unavailable keyservers
445+
if timeout 30 gpg --no-default-keyring --keyring trustedkeys.gpg \
445446
--keyserver keyserver.ubuntu.com \
446-
--recv-keys 648ACFD622F3D138 0E98404D386FA1D9
447+
--recv-keys 648ACFD622F3D138 0E98404D386FA1D9 2>&1 | grep -v "not changed\|unchanged\|not checked" || true; then
448+
echo "::notice::GPG keys imported successfully"
449+
else
450+
echo "::warning::Some GPG keys could not be imported from keyserver (non-critical)"
451+
fi
447452
448453
- name: "Prepare machine"
449454
id: preparing
@@ -460,11 +465,18 @@ jobs:
460465
[[ "${SIGNATURES}" == "ignore" ]] && APTLY_CONF+="-ignore-signatures "
461466
462467
# Read existing releases and create folder structure
468+
rm -rf build/output/{debs,debs-beta}
463469
ALL_RELEASES=($(grep -rw build/config/distributions/*/support -ve 'eos' | cut -d"/" -f4 ))
464470
for i in ${ALL_RELEASES[@]}; do
465471
mkdir -p build/output/{debs,debs-beta}/${i}
466472
mkdir -p build/output/{debs,debs-beta}/extra/${i}-utils
467473
mkdir -p build/output/{debs,debs-beta}/extra/${i}-desktop
474+
# Add example.deb to each subfolder to ensure empty repos are published
475+
for repo_root in build/output/debs build/output/debs-beta; do
476+
cp build/tools/repository/example.deb ${repo_root}/${i}/
477+
cp build/tools/repository/example.deb ${repo_root}/extra/${i}-utils/
478+
cp build/tools/repository/example.deb ${repo_root}/extra/${i}-desktop/
479+
done
468480
done
469481
470482
needs_qemu='${{ matrix.needs_qemu }}'
@@ -507,6 +519,7 @@ jobs:
507519
. os/external/${{ matrix.name }}.conf
508520
509521
SOURCE="temp/"
522+
rm -rf ${SOURCE}
510523
mkdir -p ${SOURCE}
511524
512525
# Get current version from Armbian repository
@@ -521,13 +534,7 @@ jobs:
521534
522535
# Get version from main repository
523536
BEFORE_VERSION=""
524-
525-
# Skip version check if SKIP_VERSION_CHECK is enabled (for force repopulation)
526-
if [[ "${{ inputs.SKIP_VERSION_CHECK }}" == "true" ]]; then
527-
echo "::notice::SKIP_VERSION_CHECK enabled - forcing upload regardless of existing versions"
528-
BEFORE_VERSION="0"
529-
else
530-
# Try main component, desktop component, then extra component
537+
# Try main component, desktop component, then extra component
531538
for repo_component in "main" "${{ matrix.release }}-desktop" "${{ matrix.release }}-utils"; do
532539
# Build URL for Packages index
533540
CURRENT_PACKAGES_URL="${PACKAGES_URL}/dists/${{ matrix.release }}/${repo_component}/binary-${{ matrix.arch }}/Packages.gz"
@@ -554,10 +561,9 @@ jobs:
554561
echo "::warning::Could not find version for $PKG in repository, assuming new package"
555562
BEFORE_VERSION="0"
556563
fi
557-
fi
558564
559565
echo "BEFORE_VERSION=${BEFORE_VERSION}" >> $GITHUB_OUTPUT
560-
566+
561567
# Download packages based on method
562568
if [[ ${METHOD} == gh ]]; then
563569
# GitHub release download method
@@ -833,30 +839,37 @@ jobs:
833839
# Upload packages directly to remote storage to avoid race conditions
834840
# This prevents parallel jobs from overwriting each other's packages
835841
# Note: StrictHostKeychecking=no is used here; consider using proper known_hosts in production
836-
842+
# Always upload to debs-beta for external packages
837843
if [[ ${TARGET} == main ]]; then
838-
# Upload to main repository directories
839-
if grep -qE 'B' <<< "$REPOSITORY"; then
844+
# Upload to main repository directory
845+
find "$SOURCE" -type f -name "*.deb" -exec \
846+
rsync -e "ssh -o StrictHostKeychecking=no -p ${{ secrets.HOST_UPLOAD_PORT }}" \
847+
-arvc --mkpath --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:${UPLOAD_PATH}/debs-beta/ \;
848+
849+
# Upload to debs only if version is newer than what's in repository
850+
# This is checked by comparing BEFORE_VERSION from the repository
851+
if dpkg --compare-versions "$AFTER_VERSION" gt "$BEFORE_VERSION"; then
852+
echo "::notice::Newer version $AFTER_VERSION found, uploading to debs"
840853
find "$SOURCE" -type f -name "*.deb" -exec \
841854
rsync -e "ssh -o StrictHostKeychecking=no -p ${{ secrets.HOST_UPLOAD_PORT }}" \
842-
-arvc --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:storage/debs-beta/ \;
843-
fi
844-
if grep -qE 'S' <<< "$REPOSITORY"; then
845-
find "$SOURCE" -type f -name "*.deb" -exec \
846-
rsync -e "ssh -o StrictHostKeychecking=no -p ${{ secrets.HOST_UPLOAD_PORT }}" \
847-
-arvc --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:storage/debs/ \;
855+
-arvc --mkpath --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:${UPLOAD_PATH}/debs/ \;
856+
else
857+
echo "::notice::Version $AFTER_VERSION already exists in debs (has $BEFORE_VERSION), skipping upload"
848858
fi
849859
else
850-
# Upload to specific release directories
851-
if grep -qE 'B' <<< "$REPOSITORY"; then
860+
# Upload to specific release directory in debs-beta
861+
find "$SOURCE" -type f -name "*.deb" -exec \
862+
rsync -e "ssh -o StrictHostKeychecking=no -p ${{ secrets.HOST_UPLOAD_PORT }}" \
863+
-arvc --mkpath --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:${UPLOAD_PATH}/debs-beta/extra/${{ matrix.release }}-${TARGET}/ \;
864+
865+
# Upload to debs only if version is newer than what's in repository
866+
if dpkg --compare-versions "$AFTER_VERSION" gt "$BEFORE_VERSION"; then
867+
echo "::notice::Newer version $AFTER_VERSION found, uploading to debs/extra"
852868
find "$SOURCE" -type f -name "*.deb" -exec \
853869
rsync -e "ssh -o StrictHostKeychecking=no -p ${{ secrets.HOST_UPLOAD_PORT }}" \
854-
-arvc --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:storage/debs-beta/extra/${{ matrix.release }}-${TARGET}/ \;
855-
fi
856-
if grep -qE 'S' <<< "$REPOSITORY"; then
857-
find "$SOURCE" -type f -name "*.deb" -exec \
858-
rsync -e "ssh -o StrictHostKeychecking=no -p ${{ secrets.HOST_UPLOAD_PORT }}" \
859-
-arvc --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:storage/debs/extra/${{ matrix.release }}-${TARGET}/ \;
870+
-arvc --mkpath --ignore-existing {} ${{ secrets.HOST_UPLOAD_USER }}@${{ secrets.HOST_UPLOAD }}:${UPLOAD_PATH}/debs/extra/${{ matrix.release }}-${TARGET}/ \;
871+
else
872+
echo "::notice::Version $AFTER_VERSION already exists in debs (has $BEFORE_VERSION), skipping upload"
860873
fi
861874
fi
862875

‎.github/workflows/infrastructure-repository-update.yml‎

Lines changed: 94 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,29 @@ jobs:
4040
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
4141
TEAM: "Release manager"
4242

43-
Copying:
43+
external:
44+
name: "Download external"
4445
needs: Check
46+
uses: armbian/armbian.github.io/.github/workflows/infrastructure-download-external.yml@main
47+
with:
48+
ENABLED: ${{ inputs.download_external != false || github.event.client_payload.download_external != false }}
49+
SKIP_VERSION_CHECK: false
50+
ACCESS_NAME: armbian
51+
BUILD_RUNNER: "docker"
52+
HOST_DEPLOY: "repo.armbian.com"
53+
PURGE: ${{ inputs.purge_external || false }}
54+
secrets:
55+
GPG_KEY1: ${{ secrets.GPG_KEY3 }}
56+
GPG_KEY2: ${{ secrets.GPG_KEY4 }}
57+
ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
58+
KEY_UPLOAD: ${{ secrets.KEY_UPLOAD }}
59+
HOST_UPLOAD: ${{ secrets.HOST_UPLOAD }}
60+
HOST_UPLOAD_USER: ${{ secrets.HOST_UPLOAD_USER }}
61+
HOST_UPLOAD_PORT: ${{ secrets.HOST_UPLOAD_PORT }}
62+
KNOWN_HOSTS_ARMBIAN_UPLOAD: ${{ secrets.KNOWN_HOSTS_ARMBIAN_UPLOAD }}
63+
64+
Copying:
65+
needs: external
4566
runs-on: repository
4667
steps:
4768

@@ -131,31 +152,79 @@ jobs:
131152
;;
132153
esac
133154
134-
external:
135-
name: "Download external"
155+
# Always sync external
156+
if [ -d "${INCOMING_PATH}/external/debs" ]; then
157+
rsync -av \
158+
--include='*/' \
159+
--include='*.deb' \
160+
--exclude='*' \
161+
--omit-dir-times --no-perms --no-group \
162+
"${INCOMING_PATH}/external/debs/" "${STORAGE_PATH}/debs/" \
163+
2>&1 | tee -a "$GITHUB_STEP_SUMMARY" || \
164+
echo "Warning: Some files/attrs were not transferred (code 23)" >> "$GITHUB_STEP_SUMMARY"
165+
fi
166+
if [ -d "${INCOMING_PATH}/external/debs-beta" ]; then
167+
rsync -av \
168+
--include='*/' \
169+
--include='*.deb' \
170+
--exclude='*' \
171+
--omit-dir-times --no-perms --no-group \
172+
"${INCOMING_PATH}/external/debs-beta/" "${STORAGE_PATH}/debs-beta/" \
173+
2>&1 | tee -a "$GITHUB_STEP_SUMMARY" || \
174+
echo "Warning: Some files/attrs were not transferred (code 23)" >> "$GITHUB_STEP_SUMMARY"
175+
fi
176+
177+
prepare-repos:
178+
name: "Prepare repositories"
136179
needs: Copying
137-
uses: armbian/armbian.github.io/.github/workflows/infrastructure-download-external.yml@main
138-
with:
139-
ENABLED: ${{ inputs.download_external != false || github.event.client_payload.download_external != false }}
140-
SKIP_VERSION_CHECK: false
141-
ACCESS_NAME: armbian
142-
BUILD_RUNNER: "ubuntu-latest"
143-
HOST_DEPLOY: "repo.armbian.com"
144-
PURGE: ${{ inputs.purge_external || false }}
145-
secrets:
146-
GPG_KEY1: ${{ secrets.GPG_KEY3 }}
147-
GPG_KEY2: ${{ secrets.GPG_KEY4 }}
148-
ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
149-
KEY_UPLOAD: ${{ secrets.KEY_UPLOAD }}
150-
HOST_UPLOAD: ${{ secrets.HOST_UPLOAD }}
151-
HOST_UPLOAD_USER: ${{ secrets.HOST_UPLOAD_USER }}
152-
HOST_UPLOAD_PORT: ${{ secrets.HOST_UPLOAD_PORT }}
153-
KNOWN_HOSTS_ARMBIAN_UPLOAD: ${{ secrets.KNOWN_HOSTS_ARMBIAN_UPLOAD }}
180+
runs-on: repository
181+
steps:
182+
- name: Checkout build repository
183+
uses: actions/checkout@v6
184+
with:
185+
repository: armbian/build
186+
path: build
187+
clean: false
188+
189+
- name: Checkout armbian.github.io repository
190+
uses: actions/checkout@v6
191+
with:
192+
repository: armbian/armbian.github.io
193+
path: armbian-github-io
194+
clean: false
195+
196+
- name: Copy example.deb to repository subfolders
197+
run: |
198+
199+
# Get all supported releases from build config
200+
ALL_RELEASES=($(grep -rw build/config/distributions/*/support -ve 'eos' | cut -d"/" -f4))
201+
echo "Found ${#ALL_RELEASES[@]} releases: ${ALL_RELEASES[*]}" | tee -a $GITHUB_STEP_SUMMARY
202+
203+
204+
sudo cp build/tools/repository/example.deb "${{ env.STORAGE_PATH }}/debs/$RELEASE/"
205+
sudo cp build/tools/repository/example.deb "${{ env.STORAGE_PATH }}/debs-beta/$RELEASE/"
154206
207+
# Copy example.deb to all subfolders for debs
208+
for RELEASE in "${ALL_RELEASES[@]}"; do
209+
echo "Copying example.deb for debs/$RELEASE and subfolders" | tee -a $GITHUB_STEP_SUMMARY
210+
mkdir -p "${{ env.STORAGE_PATH }}/debs/extra/$RELEASE-utils"
211+
mkdir -p "${{ env.STORAGE_PATH }}/debs/extra/$RELEASE-desktop"
212+
sudo cp build/tools/repository/example.deb "${{ env.STORAGE_PATH }}/debs/extra/$RELEASE-utils/"
213+
sudo cp build/tools/repository/example.deb "${{ env.STORAGE_PATH }}/debs/extra/$RELEASE-desktop/"
214+
done
215+
216+
# Copy example.deb to all subfolders for debs-beta
217+
for RELEASE in "${ALL_RELEASES[@]}"; do
218+
echo "Copying example.deb for debs-beta/$RELEASE and subfolders" | tee -a $GITHUB_STEP_SUMMARY
219+
mkdir -p "${{ env.STORAGE_PATH }}/debs-beta/extra/$RELEASE-utils"
220+
mkdir -p "${{ env.STORAGE_PATH }}/debs-beta/extra/$RELEASE-desktop"
221+
sudo cp build/tools/repository/example.deb "${{ env.STORAGE_PATH }}/debs-beta/extra/$RELEASE-utils/"
222+
sudo cp build/tools/repository/example.deb "${{ env.STORAGE_PATH }}/debs-beta/extra/$RELEASE-desktop/"
223+
done
155224
156225
fix-permissions:
157226
name: "Fix permissions"
158-
needs: external
227+
needs: prepare-repos
159228
runs-on: repository
160229
strategy:
161230
matrix:
@@ -333,7 +402,7 @@ jobs:
333402
with:
334403
repository: armbian/build
335404
fetch-depth: 1
336-
ref: main
405+
ref: always
337406
clean: false
338407

339408
- name: "Build repository ${{ matrix.repository.name }}"
@@ -412,7 +481,7 @@ jobs:
412481
with:
413482
repository: armbian/build
414483
clean: false
415-
ref: main
484+
ref: always
416485
fetch-depth: 1
417486
path: build
418487

@@ -534,7 +603,7 @@ jobs:
534603
uses: actions/checkout@v6
535604
with:
536605
repository: armbian/build
537-
ref: main
606+
ref: always
538607
fetch-depth: 1
539608
clean: false
540609

@@ -646,7 +715,7 @@ jobs:
646715
uses: actions/checkout@v6
647716
with:
648717
repository: armbian/build
649-
ref: main
718+
ref: always
650719
fetch-depth: 1
651720
clean: false
652721

0 commit comments

Comments
 (0)