tools, .github/workflows: harden EEST fixture download and warm its cache out-of-band (erigontech#21674)

## Problem

The `eest-spec-*` checks failed on a recent run
([example](https://github.com/erigontech/erigon/actions/runs/27119504237))
with:

```
curl: (22) The requested URL returned error: 504   ← x4
make: *** [Makefile:271: test-fixtures-eest] Error 22
```

Two distinct issues surfaced:

1. **Transient download failures.** On a cache miss,
`tools/test-fixtures.sh` downloads fixture tarballs directly from the
GitHub release CDN. It retried only `--retry 3` over a fixed ~6s window,
which couldn't ride out a burst of HTTP 504s.
2. **Cache-entry churn + cold misses.** `test-eest-spec.yml` used
`actions/cache` (restore **and** save), so every PR ref
(`refs/pull/<N>/merge`) saved its own ~1.6 GB copy of identical fixture
tarballs. Sibling PR refs can't read each other's caches, so this gave
no shared-hit benefit and churned the shared cache budget via LRU. There
was also no default-branch (`main`) entry, so every PR cold-missed in
the first place.

## Fix

- **`tools/test-fixtures.sh`** — curl exponential backoff over a
5-minute window, fail-fast connect, and DNS re-resolution each attempt
(CDN records can rotate on short TTLs):
  ```
--retry 10 --retry-all-errors --retry-delay 0 --retry-max-time 300
--connect-timeout 30 --dns-cache-timeout 0
  ```
- **`test-eest-spec.yml`** — switch to `actions/cache/restore`
(**restore-only**). PR / merge_group runs never save, so no per-ref
duplicates. A miss self-heals via `tools/test-fixtures.sh`.
- **`cache-warming-eest-fixtures.yml` (new)** — a dedicated workflow
that populates the base-branch cache, since default-branch caches are
readable from every PR. It runs on `test-fixtures.json` change + every 2
days (cron) + `workflow_dispatch`, probes with `lookup-only`, and
downloads + saves only on a real miss, fetching all four tarballs so the
entry is complete.

Design notes: key kept as `hashFiles('test-fixtures.json')`;
`cl_mainnet` intentionally excluded (owned by
`test-integration-caplin.yml`); default cache compression kept —
restore-only moves compression off the PR critical path into the
infrequent warmer (`actions/cache` exposes no public switch to disable
it — see actions/toolkit#544).

## Bootstrap note

After merge the warmer won't auto-fire (it doesn't change
`test-fixtures.json`), so until the next 2-day cron tick PRs would
cold-miss (self-healing, just slower). Run the **`Cache Warming — EEST
fixtures`** workflow once via `workflow_dispatch` right after merge to
populate the `main`-scoped cache immediately.

## Verification

- `actionlint` (bundles shellcheck) — clean on both workflows.
- `bash -n tools/test-fixtures.sh` — clean.
- YAML parses.

Author: lystopad

.github/workflows/cache-warming-eest-fixtures.yml

@@ -0,0 +1,77 @@
+name: Cache Warming — EEST fixtures
+
+# Populates the base-branch EEST fixture-tarball cache that test-eest-spec.yml
+# restores (restore-only there, so PR / merge_group runs never save their own
+# per-ref duplicates). GitHub cache scoping makes default-branch caches readable
+# from every PR and merge group, so warming on main/release is enough.
+#
+# Triggers:
+#   - push to a protected branch touching test-fixtures.json -> warm on a version bump
+#   - schedule (daily)                                       -> safety net if LRU-evicted
+#   - workflow_dispatch                                      -> manual (use once to bootstrap)
+#
+# The key matches test-eest-spec.yml exactly:
+#   test-fixtures-eest-${{ runner.os }}-${{ hashFiles('test-fixtures.json') }}
+
+on:
+  push:
+    branches:
+      - main
+      - 'release/**'
+    paths:
+      - test-fixtures.json
+  schedule:
+    - cron: '0 4 * * *'
+  workflow_dispatch:
+
+defaults:
+  run:
+    shell: bash
+
+permissions:
+  contents: read
+
+concurrency:
+  group: cache-warming-eest-fixtures-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  warm:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 1
+
+      # Probe for an exact-key hit without downloading the archive. cache-hit is
+      # true only on the exact primary key, so a version bump (new key) reports a
+      # miss even when an older entry still matches the restore-key prefix.
+      - name: Probe EEST cache
+        id: probe
+        uses: actions/cache/restore@v5
+        with:
+          path: |
+            test-fixtures-cache/eest_stable.tar.gz
+            test-fixtures-cache/eest_devnet.tar.gz
+            test-fixtures-cache/eest_benchmark.tar.gz
+            test-fixtures-cache/eest_zkevm.tar.gz
+          key: test-fixtures-eest-${{ runner.os }}-${{ hashFiles('test-fixtures.json') }}
+          lookup-only: true
+
+      # Download all four tarballs in one invocation so the single cached entry
+      # is complete (a shard run only fetches the subset it needs).
+      - name: Download EEST tarballs
+        if: steps.probe.outputs.cache-hit != 'true'
+        run: bash tools/test-fixtures.sh test-fixtures.json test-fixtures-cache eest_stable eest_devnet eest_benchmark eest_zkevm
+
+      - name: Save EEST cache
+        if: steps.probe.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v5
+        with:
+          path: |
+            test-fixtures-cache/eest_stable.tar.gz
+            test-fixtures-cache/eest_devnet.tar.gz
+            test-fixtures-cache/eest_benchmark.tar.gz
+            test-fixtures-cache/eest_zkevm.tar.gz
+          key: test-fixtures-eest-${{ runner.os }}-${{ hashFiles('test-fixtures.json') }}

.github/workflows/test-eest-spec.yml

@@ -58,13 +58,17 @@ jobs:
           ramdisk: true
           build-cache-extra-key: eest-spec-${{ matrix.shard }}
 
-      - name: Cache EEST tarballs
-        uses: actions/cache@v5
+      # Restore-only: PR / merge_group runs never SAVE, so they don't mint a
+      # per-ref ~1.6GB duplicate every run. The base-branch (main) entry is
+      # populated by cache-warming-eest-fixtures.yml; default-branch caches are
+      # readable from every PR. A miss here self-heals (tools/test-fixtures.sh
+      # re-downloads), and the warmer repopulates main.
+      - name: Restore EEST tarballs
+        uses: actions/cache/restore@v5
         with:
-          # Only cache the .tar.gz files; extracted dirs are recreated by
-          # tools/test-fixtures.sh on each run from the cached tarballs.
-          # Dedicated EEST cache so this workflow doesn't compete with
-          # test-integration-caplin.yml's cl_mainnet cache under one key.
+          # Only the .tar.gz files are cached; tools/test-fixtures.sh re-extracts
+          # them each run. cl_mainnet is intentionally excluded (owned by
+          # test-integration-caplin.yml's separate cache).
           path: |
             test-fixtures-cache/eest_stable.tar.gz
             test-fixtures-cache/eest_devnet.tar.gz

tools/test-fixtures.sh

@@ -80,7 +80,10 @@ while IFS=$'\t' read -r name url want; do
 
 	if [[ ! -f "$tar_path" ]] || [[ "$(sha256 "$tar_path")" != "$want" ]]; then
 		echo "$name: downloading from $url"
-		curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 -o "$tar_path.tmp" "$url"
+		# Ride out transient upstream 5xx (release CDN) with exponential backoff over a 5-min window.
+		curl -fsSL --retry 10 --retry-all-errors --retry-delay 0 \
+			--retry-max-time 300 --connect-timeout 30 \
+			-o "$tar_path.tmp" "$url"
 		got=$(sha256 "$tar_path.tmp")
 		if [[ "$got" != "$want" ]]; then
 			rm -f "$tar_path.tmp"