ci: move SonarCloud branch scans out of the merge queue, serialize per branch (erigontech#21669)

## Problem

SonarCloud rejects a branch analysis dated older than the branch's
latest processed one. Every merge-queue entry analyzes the target
branch, so when queue entry N's report uploads *after* entry N+1's,
SonarCloud rejects N's with *"Date of analysis cannot be older than the
date of the last known analysis"* and flips that commit's **SonarCloud
Code Analysis** check to `cancelled`.

Seen on
[`5027d259c2`](erigontech@5027d25):
three stacked queue entries (erigontech#21648, erigontech#21584, erigontech#21640) scanned
concurrently; erigontech#21584 started 6s later but uploaded 21s earlier, so
erigontech#21648's analysis was rejected as out of order. No impact on the gate
(the scan step is `continue-on-error` and the merge succeeded), but it
leaves a spurious failed check on a merged commit.

## Fix

Make it structurally impossible to submit branch analyses out of order:

- **`merge_group` runs** keep `make test-sonar-coverage` as a gate check
but **no longer scan**. They upload `coverage-test-all.out` as a 7-day
artifact.
- **A new `SonarCloud Branch Scan` workflow** (`push` on `main` +
`release/**`, matching the branches SonarCloud tracks) downloads that
artifact and runs **only the scanner** (~7 min vs ~25). The merge queue
fast-forwards the target branch to the already-tested merge commit, so
the pushed SHA equals the merge-queue run's head SHA and the artifact
maps to the commit exactly. If no artifact exists (direct push, or
expired retention), it falls back to running the tests itself.
- Branch scans are serialized per branch **FIFO** via
`concurrency.queue: max` ([GitHub Actions,
2026-05](https://github.blog/changelog/2026-05-07-github-actions-concurrency-groups-now-allow-larger-queues/)),
so every merged commit is analyzed, in order, with **no runs cancelled**
during merge bursts.
- **PR scans** (Sonar PR analyses — separate from main-branch date
ordering) and **cache-warming** runs are unchanged.

A failed analysis upload no longer touches the gate at all: scanner
failures now surface on a post-merge run, while PR runs keep exercising
the scanner so config/scanner breakage is still caught before merge.

## Notes

- `actionlint` (≤ v1.7.12, the latest release) does not know the `queue`
concurrency key yet, so `lint.yml` gets a narrow `--ignore` alongside
the existing `workspace` one. Removable once actionlint learns the key.
- `concurrency.queue: max` is ~1 month old — first real merge-burst
after this lands is where FIFO ordering gets exercised live.

## Testing

- `make lint`, `actionlint` (CI flags), and `zizmor` (repo config) all
clean; the new workflow is finding-free.
- The artifact-lookup script was shellchecked and run against live API
data: correctly resolves the merge-queue run by head SHA, filters
expired artifacts, and degrades to the test-running fallback on a
missing artifact or total API failure (never fails the job).

Author: taratorio

.github/workflows/lint.yml

@@ -33,8 +33,13 @@ jobs:
       # runner.workspace is suppressed: it was dropped from the official context
       # spec but is still available on self-hosted runners; QA workflows rely on
       # it until they are migrated to github.workspace / runner.temp.
+      # The "queue" concurrency key (GitHub, 2026-05) is newer than any released
+      # actionlint schema; drop that ignore once actionlint learns the key.
       - name: actionlint
-        run: actionlint -shellcheck '' --ignore '"workspace" is not defined in object type'
+        run: |
+          actionlint -shellcheck '' \
+            --ignore '"workspace" is not defined in object type' \
+            --ignore 'unexpected key "queue" for "concurrency" section'
 
       - name: Actions security audit (zizmor)
         run: |

.github/workflows/sonar-branch-scan.yml

@@ -0,0 +1,28 @@
+name: SonarCloud Branch Scan
+
+# SonarCloud rejects a branch analysis dated older than the branch's latest
+# processed one, so analyses must reach it in chronological order. This
+# workflow is the only submitter of branch analyses: it scans after each push,
+# serialized per branch by a FIFO concurrency queue, reusing the coverage
+# profile uploaded by the commit's merge-queue run.
+
+on:
+  push:
+    branches:
+      - main
+      - 'release/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  queue: max
+
+permissions:
+  actions: read
+  contents: read
+
+jobs:
+  sonar:
+    uses: ./.github/workflows/sonar.yml
+    with:
+      scan-only: true
+    secrets: inherit

.github/workflows/sonar.yml

@@ -8,6 +8,10 @@ on:
         description: "Compile test binaries only; skip test execution (for cache warming runs)"
         type: boolean
         default: false
+      scan-only:
+        description: "Reuse the coverage artifact from this commit's merge-queue run instead of running tests (for post-merge branch scans)"
+        type: boolean
+        default: false
 
 defaults:
   run:
@@ -34,12 +38,59 @@ jobs:
           cleanup-space: true
           ramdisk: true
 
+      # The merge queue fast-forwards the target branch to the already-tested
+      # merge commit, so the pushed SHA equals a merge-queue run's head SHA and
+      # that run's coverage artifact corresponds to this commit exactly.
+      - name: Locate merge-queue coverage artifact
+        id: coverage-artifact
+        if: inputs.scan-only
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          found=false
+          run_id=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs?head_sha=${GITHUB_SHA}&event=merge_group" \
+            --jq '[.workflow_runs[] | select(.path == ".github/workflows/ci-gate.yml")][0].id // empty' 2>/dev/null) || run_id=""
+          if [ -n "$run_id" ]; then
+            count=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${run_id}/artifacts?name=sonar-coverage" \
+              --jq '[.artifacts[] | select(.expired | not)] | length' 2>/dev/null) || count=0
+            if [ "${count:-0}" -gt 0 ]; then
+              found=true
+            fi
+          fi
+          if [ "$found" = "false" ]; then
+            echo "::warning::No merge-queue coverage artifact for ${GITHUB_SHA}; running tests with coverage in this run"
+          fi
+          {
+            echo "found=$found"
+            echo "run_id=$run_id"
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Download merge-queue coverage artifact
+        if: inputs.scan-only && steps.coverage-artifact.outputs.found == 'true'
+        uses: actions/download-artifact@v8
+        with:
+          name: sonar-coverage
+          run-id: ${{ steps.coverage-artifact.outputs.run_id }}
+          github-token: ${{ github.token }}
+
       - name: Run tests with coverage
-        if: '!inputs.cache-warming-only'
+        if: "!inputs.cache-warming-only && (!inputs.scan-only || steps.coverage-artifact.outputs.found != 'true')"
         env:
           SKIP_FLAKY_TESTS: 'true'
         run: make test-sonar-coverage
 
+      # Merge-queue runs produce the coverage profile but leave the scan to the
+      # push-triggered SonarCloud Branch Scan workflow, which submits branch
+      # analyses one at a time so SonarCloud never rejects them as out of order.
+      - name: Upload coverage artifact for the post-merge scan
+        if: "!inputs.cache-warming-only && github.event_name == 'merge_group'"
+        uses: actions/upload-artifact@v7
+        with:
+          name: sonar-coverage
+          path: coverage-test-all.out
+          retention-days: 7
+          if-no-files-found: error
+
       - name: Build test binaries (cache warming)
         if: inputs.cache-warming-only
         run: go test -run=^$ -cover ./...
@@ -99,7 +150,7 @@ jobs:
       # otherwise download from scanner.sonarcloud.io (a 403-prone CDN);
       # if the image ever drops it, the scanner falls back to downloading.
       - name: Use preinstalled JDK for SonarCloud scanner
-        if: '!inputs.cache-warming-only'
+        if: "!inputs.cache-warming-only && github.event_name != 'merge_group'"
         run: |
           if [ -n "${JAVA_HOME_21_X64}" ] && [ -x "${JAVA_HOME_21_X64}/bin/java" ]; then
             echo "SONAR_SCANNER_SKIP_JRE_PROVISIONING=true" >> "$GITHUB_ENV"
@@ -111,7 +162,7 @@ jobs:
       # scanner.sonarcloud.io. A miss (engine version rotated since the last
       # base-branch push) falls back to downloading plus the retry below.
       - name: Restore SonarCloud scanner engine cache
-        if: '!inputs.cache-warming-only'
+        if: "!inputs.cache-warming-only && github.event_name != 'merge_group'"
         uses: actions/cache/restore@v5
         with:
           path: ~/.sonar/cache
@@ -123,19 +174,19 @@ jobs:
       # rides out short blips instead of failing the gate.
       - name: SonarCloud scan
         id: scan
-        if: '!inputs.cache-warming-only'
+        if: "!inputs.cache-warming-only && github.event_name != 'merge_group'"
         continue-on-error: true
         uses: SonarSource/sonarqube-scan-action@v8
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
 
       - name: Wait before SonarCloud scan retry
-        if: "!inputs.cache-warming-only && steps.scan.outcome == 'failure'"
+        if: steps.scan.outcome == 'failure'
         run: sleep 90
 
       - name: SonarCloud scan (retry)
-        if: "!inputs.cache-warming-only && steps.scan.outcome == 'failure'"
+        if: steps.scan.outcome == 'failure'
         uses: SonarSource/sonarqube-scan-action@v8
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}