Skip to content

Commit e2d6e70

Browse files
castrojoCopilot
andcommitted
feat(ci): ETag+cache nightly check every 12h, document cache rule
- Rewrite update-mozilla-nightly.yml to HEAD-check ETags before downloading; skip 100MB+ downloads if upstream hasn't changed - Cache ETag files with actions/cache (split restore/save) so the check is free on no-change runs - Schedule every 12h instead of daily - Document 'always use actions/cache' as rule 7 in skills/pipeline.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent fdd7041 commit e2d6e70

File tree

2 files changed

+66
-42
lines changed

2 files changed

+66
-42
lines changed

.github/workflows/update-mozilla-nightly.yml

Lines changed: 64 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: Update Mozilla Nightly sha256s
22

33
on:
44
schedule:
5-
- cron: '0 6 * * 1' # Monday 6am UTC
5+
- cron: '0 */12 * * *' # Every 12 hours
66
workflow_dispatch:
77

88
permissions:
@@ -16,52 +16,74 @@ jobs:
1616
- name: Checkout
1717
uses: actions/checkout@v4
1818

19+
- name: Restore cached ETags
20+
uses: actions/cache/restore@v4
21+
with:
22+
path: /tmp/nightly-etags
23+
key: mozilla-nightly-etags-${{ github.run_id }}
24+
restore-keys: mozilla-nightly-etags-
25+
26+
- name: Check ETags and download if changed
27+
id: check
28+
run: |
29+
mkdir -p /tmp/nightly-etags
30+
31+
FF_X86_URL="https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-150.0a1.en-US.linux-x86_64.tar.xz"
32+
FF_ARM_URL="https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-150.0a1.en-US.linux-aarch64.tar.xz"
33+
TB_X86_URL="https://download-installer.cdn.mozilla.net/pub/thunderbird/nightly/latest-comm-central/thunderbird-150.0a1.en-US.linux-x86_64.tar.xz"
34+
35+
CHANGED=false
36+
37+
check_and_download() {
38+
local name="$1" url="$2" etag_file="/tmp/nightly-etags/${1}.etag"
39+
local new_etag old_etag
40+
new_etag=$(curl -sI "$url" | grep -i '^etag:' | tr -d '\r' | awk '{print $2}')
41+
old_etag=$(cat "$etag_file" 2>/dev/null || echo "")
42+
if [ "$new_etag" = "$old_etag" ] && [ -n "$new_etag" ]; then
43+
echo "$name: ETag unchanged ($new_etag), skipping download"
44+
echo "changed_${name}=false" >> "$GITHUB_OUTPUT"
45+
else
46+
echo "$name: ETag changed ($old_etag -> $new_etag), downloading..."
47+
curl -sL --output "/tmp/${name}.tar.xz" "$url"
48+
echo "$new_etag" > "$etag_file"
49+
echo "changed_${name}=true" >> "$GITHUB_OUTPUT"
50+
fi
51+
}
52+
53+
check_and_download ff_x86 "$FF_X86_URL"
54+
check_and_download ff_arm "$FF_ARM_URL"
55+
check_and_download tb_x86 "$TB_X86_URL"
56+
1957
- name: Update firefox-nightly x86_64 sha256
20-
id: ff-x86_64
58+
if: steps.check.outputs.changed_ff_x86 == 'true'
2159
run: |
22-
URL="https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-150.0a1.en-US.linux-x86_64.tar.xz"
23-
curl -sL --output /tmp/firefox-x86_64.tar.xz "$URL"
24-
NEW_SHA256=$(sha256sum /tmp/firefox-x86_64.tar.xz | cut -d' ' -f1)
25-
OLD_SHA256=$(grep -A1 'firefox-150.0a1.en-US.linux-x86_64.tar.xz' flatpaks/firefox-nightly/manifest.yaml | grep 'sha256:' | awk '{print $2}')
26-
if [ "$NEW_SHA256" != "$OLD_SHA256" ]; then
27-
sed -i "/url:.*firefox-150.0a1.en-US.linux-x86_64.tar.xz/{n;s/sha256:.*/sha256: $NEW_SHA256/}" flatpaks/firefox-nightly/manifest.yaml
28-
echo "changed=true" >> "$GITHUB_OUTPUT"
29-
else
30-
echo "changed=false" >> "$GITHUB_OUTPUT"
31-
fi
60+
NEW=$(sha256sum /tmp/ff_x86.tar.xz | cut -d' ' -f1)
61+
sed -i "/url:.*firefox-150.0a1.en-US.linux-x86_64.tar.xz/{n;s/sha256:.*/sha256: $NEW/}" flatpaks/firefox-nightly/manifest.yaml
3262
3363
- name: Update firefox-nightly aarch64 sha256
34-
id: ff-aarch64
64+
if: steps.check.outputs.changed_ff_arm == 'true'
3565
run: |
36-
URL="https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-150.0a1.en-US.linux-aarch64.tar.xz"
37-
curl -sL --output /tmp/firefox-aarch64.tar.xz "$URL"
38-
NEW_SHA256=$(sha256sum /tmp/firefox-aarch64.tar.xz | cut -d' ' -f1)
39-
OLD_SHA256=$(grep -A1 'firefox-150.0a1.en-US.linux-aarch64.tar.xz' flatpaks/firefox-nightly/manifest.yaml | grep 'sha256:' | awk '{print $2}')
40-
if [ "$NEW_SHA256" != "$OLD_SHA256" ]; then
41-
sed -i "/url:.*firefox-150.0a1.en-US.linux-aarch64.tar.xz/{n;s/sha256:.*/sha256: $NEW_SHA256/}" flatpaks/firefox-nightly/manifest.yaml
42-
echo "changed=true" >> "$GITHUB_OUTPUT"
43-
else
44-
echo "changed=false" >> "$GITHUB_OUTPUT"
45-
fi
66+
NEW=$(sha256sum /tmp/ff_arm.tar.xz | cut -d' ' -f1)
67+
sed -i "/url:.*firefox-150.0a1.en-US.linux-aarch64.tar.xz/{n;s/sha256:.*/sha256: $NEW/}" flatpaks/firefox-nightly/manifest.yaml
4668
4769
- name: Update thunderbird-nightly x86_64 sha256
48-
id: tb-x86_64
70+
if: steps.check.outputs.changed_tb_x86 == 'true'
4971
run: |
50-
URL="https://download-installer.cdn.mozilla.net/pub/thunderbird/nightly/latest-comm-central/thunderbird-150.0a1.en-US.linux-x86_64.tar.xz"
51-
curl -sL --output /tmp/thunderbird-x86_64.tar.xz "$URL"
52-
NEW_SHA256=$(sha256sum /tmp/thunderbird-x86_64.tar.xz | cut -d' ' -f1)
53-
OLD_SHA256=$(grep -A1 'thunderbird-150.0a1.en-US.linux-x86_64.tar.xz' flatpaks/thunderbird-nightly/manifest.yaml | grep 'sha256:' | awk '{print $2}')
54-
if [ "$NEW_SHA256" != "$OLD_SHA256" ]; then
55-
sed -i "/url:.*thunderbird-150.0a1.en-US.linux-x86_64.tar.xz/{n;s/sha256:.*/sha256: $NEW_SHA256/}" flatpaks/thunderbird-nightly/manifest.yaml
56-
echo "changed=true" >> "$GITHUB_OUTPUT"
57-
else
58-
echo "changed=false" >> "$GITHUB_OUTPUT"
59-
fi
72+
NEW=$(sha256sum /tmp/tb_x86.tar.xz | cut -d' ' -f1)
73+
sed -i "/url:.*thunderbird-150.0a1.en-US.linux-x86_64.tar.xz/{n;s/sha256:.*/sha256: $NEW/}" flatpaks/thunderbird-nightly/manifest.yaml
74+
75+
- name: Save ETags to cache
76+
uses: actions/cache/save@v4
77+
if: always()
78+
with:
79+
path: /tmp/nightly-etags
80+
key: mozilla-nightly-etags-${{ github.run_id }}
6081

6182
- name: Open PR if sha256s changed
62-
id: pr
6383
env:
64-
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
84+
# NOTE: PRs opened with GITHUB_TOKEN do not trigger pull_request CI.
85+
# Add a PAT as NIGHTLY_UPDATE_TOKEN to make builds trigger automatically.
86+
GH_TOKEN: ${{ secrets.NIGHTLY_UPDATE_TOKEN || secrets.GITHUB_TOKEN }}
6587
run: |
6688
git config user.name "github-actions[bot]"
6789
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
@@ -71,16 +93,16 @@ jobs:
7193
exit 0
7294
fi
7395
74-
BRANCH="chore/nightly-sha256-$(date -u +%Y%m%d)"
96+
BRANCH="chore/nightly-sha256-$(date -u +%Y%m%d-%H%M)"
7597
git checkout -b "$BRANCH"
7698
git add flatpaks/firefox-nightly/manifest.yaml flatpaks/thunderbird-nightly/manifest.yaml
77-
printf 'chore(nightly): update Mozilla nightly sha256s\n\nAuto-refresh sha256 for firefox-nightly and thunderbird-nightly.\nMozilla rebuilds nightly at the same URL daily; version string stays\n150.0a1 so Renovate cannot track this.\n\nCo-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>\n' > /tmp/nightly-commit-msg.txt
78-
git commit -F /tmp/nightly-commit-msg.txt
99+
printf 'chore(nightly): update Mozilla nightly sha256s\n\nAuto-refresh sha256 for firefox-nightly and thunderbird-nightly.\nMozilla rebuilds nightly at the same URL; ETag changed so new build\nis available. Renovate cannot track this.\n\nCo-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>\n' > /tmp/commit-msg.txt
100+
git commit -F /tmp/commit-msg.txt
79101
git push origin "$BRANCH"
80102
81103
gh pr create \
82104
--title "chore(nightly): update Mozilla nightly sha256s $(date -u +%Y-%m-%d)" \
83-
--body "Auto-refresh sha256 for firefox-nightly and thunderbird-nightly. Mozilla rebuilds nightly at the same URL daily; version string stays 150.0a1 so Renovate cannot track this." \
105+
--body "ETag-triggered sha256 refresh for firefox-nightly and/or thunderbird-nightly." \
84106
--base main \
85-
--head "$BRANCH"
86-
gh pr merge "$BRANCH" --auto --squash
107+
--head "$BRANCH" || true
108+
gh pr merge "$BRANCH" --auto --squash || true

skills/pipeline.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -481,6 +481,8 @@ Simultaneous pushes create concurrent runs that fight for the same runners and w
481481

482482
6. **Concurrency group awareness.** `build.yml` has a `concurrency` group per app — a new push for the same app cancels an in-progress run. This is intentional for feature branches but undesirable for `main`. Avoid pushing rapidly in succession on main.
483483

484+
7. **Always use `actions/cache`.** Any workflow that downloads large files, computes hashes, or repeats expensive operations should cache intermediate results. Use ETags or content hashes as cache keys so the cache is only busted when upstream actually changes. Prefer `actions/cache/restore` + `actions/cache/save` (split) over the combined `actions/cache` so you can save even on failure (`if: always()`). Example pattern used in `update-mozilla-nightly.yml`: cache ETag files with key `<prefix>-${{ github.run_id }}` and `restore-keys: <prefix>-` so each run saves fresh ETags while always restoring the most recent prior run's values.
485+
484486
**Quick check command:**
485487
```bash
486488
gh run list --repo projectbluefin/testhub --workflow=build.yml --limit 5 \

0 commit comments

Comments
 (0)