Skip to content

PS-11078: unify per-PR + nightly arm64 builds into build.yml#5972

Merged
nogueiraanderson merged 5 commits into
8.0from
PS-11078-arm64-nightly
May 27, 2026
Merged

PS-11078: unify per-PR + nightly arm64 builds into build.yml#5972
nogueiraanderson merged 5 commits into
8.0from
PS-11078-arm64-nightly

Conversation

@nogueiraanderson

@nogueiraanderson nogueiraanderson commented May 22, 2026

Copy link
Copy Markdown
Contributor

Feature

  • Single .github/workflows/build.yml on 8.0 handling three event paths: pull_request to 8.0 (Debug + main.1st MTR, Cirrus per-PR replacement), schedule '0 1 * * *' (RelWithDebInfo + binlog_nogtid MTR, Cirrus nightly replacement), and workflow_dispatch (manual with build_type + debug_keep_vm inputs).

Why

  • Consolidates build-arm64.yml + build-arm64-nightly.yml into one file per @inikep's 2026-05-27 review ask; dispatch job picks BUILD_TYPE/MTR_SUITE/CCACHE_MAXSIZE from github.event_name so per-PR + nightly share one build pipeline.
  • This file is 8.0-only; trunk and 8.4 adopt the same shape in follow-up PRs (owned by @inikep) with their per-branch cmake/apt deltas.
  • x86_64 nightly RelWithDebInfo can plug in as a sibling job reusing the same dispatch outputs; intentionally not wired in this PR per "let's see how arm64 nightly behaves first".

Tickets

Replaces the Cirrus `(arm64) gcc RelWithDebInfo [Noble]` task on 8.0,
trunk, and 8.4 before Cirrus shuts down 2026-06-01.

Centralised on the 8.0 default branch because GHA `schedule:` triggers
only fire from the default branch. Three cron entries dispatch to the
corresponding ref via the `pickbranch` job; the build step gates Boost
cache, KEYRING_VAULT vs COMPONENT_KEYRING_VAULT, READLINE vs EDITLINE,
and WITH_CURL on the target branch.

MTR suite: `binlog_nogtid` per Przemek 2026-05-18:
"I would keep binlog_nogtid for RelWithDebInfo cron."

Manual `workflow_dispatch` exposes a branch selector for ad-hoc runs.
The nightly workflow had no pull_request trigger, so opening this PR
could not exercise the new file. The other PR's CI green came from
build-arm64.yml firing incidentally (because PR target is 8.0), not
from validating the nightly file's RelWithDebInfo + binlog_nogtid +
pickbranch logic.

Add a path-filtered pull_request trigger so the workflow self-tests
when the file itself changes. pickbranch handles pull_request by
building against `github.base_ref` (the PR's target branch), exercising
the branch-aware cmake conditional for that target.

This stays in the merged file as a permanent safety net: future edits
to build-arm64-nightly.yml will self-validate before merge.
Three flakes today (5974 first, 5972 first, 5972 rerun) all failed at
the same step: Hetzner VM provisions fine, but the runner agent never
calls home within the 10-min `runner_wait` budget. Codex diagnosis
2026-05-22: cloud-init bootstrap (apt update/install, runner release
download, config.sh) hits transient slowness; default budget too tight.

- runner_version: '2.334.0' (was implicit 'latest', fetched on every boot)
- runner_wait: '120' (= 120x10s = 20min, was default 10min)
- pre_runner_script: retry apt 3x with 15/30/45s backoff
- concurrency: pull_request gets PR-scoped group + cancel-in-progress
  (was global, never-cancel). Cron + workflow_dispatch stay protected.

Followup (separate PR): same hardening on build-arm64.yml; ship a
debug escape hatch that preserves /var/log/cloud-init-output.log on
create-runner failure.
Tonight's 4th create-runner failure (run 26314796152) was not transient
apt slowness or fixed by pinning runner_version=2.334.0; Hetzner appears
degraded (cax41 fully unavailable, 429 burst on the token, repeated
runner-registration timeouts). The pin was a speculative variable that
shouldn't stay until we have VM-side evidence.

Reverts:
- runner_version (was '2.334.0', back to action default 'latest')

Keeps (cheap, useful regardless):
- runner_wait: '120' (20min budget)
- pre_runner_script: apt retry 3x with backoff

Adds:
- workflow_dispatch input `debug_keep_vm` (boolean, default false)
- Step `Preserve VM for manual diagnosis (on failure, debug-only)` in
  create-runner that prints VM IP + SSH command + cleanup snippet to the
  GHA step summary when debug_keep_vm is true and create-runner failed
- delete-runner skip condition: when debug_keep_vm && create-runner failure,
  leave the VM alive for SSH diagnosis. orphan-sweep reaps after 6h.

Tomorrow morning workflow:
1. Trigger workflow_dispatch with debug_keep_vm=true
2. Wait for create-runner failure (or success)
3. If fail: SSH in with key 107239874, grab /var/log/cloud-init-output.log
   and /actions-runner/_diag/, then DELETE the VM via Hetzner API
@nogueiraanderson nogueiraanderson changed the title PS-11078: nightly arm64 RelWithDebInfo + binlog_nogtid (centralised on 8.0) PS-11078: nightly arm64 RelWithDebInfo + binlog_nogtid on 8.0 May 23, 2026
git submodule update --init --force --depth=256
git submodule

- name: System and compiler info

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be called Compiler and cmake info now.

@inikep

inikep commented May 25, 2026

Copy link
Copy Markdown
Collaborator

Looks good. Is it possible to have the same .yml file for nightly jobs and jobs triggered by PR ? It would help a lot with maintenance.

- New build.yml: dispatch job picks BUILD_TYPE/MTR/CCACHE_MAXSIZE per event
  (pull_request -> Debug+main.1st, schedule '0 1 * * *' -> RelWithDebInfo+binlog_nogtid,
  workflow_dispatch -> input.build_type)
- Drop pickbranch + branch-aware cmake/apt; this file is 8.0-only per Przemek 2026-05-27
- Rename step "System and compiler info" -> "Compiler and cmake info"
- x86_64 nightly intentionally not wired; sibling job can reuse dispatch outputs later
- Delete build-arm64.yml + build-arm64-nightly.yml (folded in)
@nogueiraanderson nogueiraanderson changed the title PS-11078: nightly arm64 RelWithDebInfo + binlog_nogtid on 8.0 PS-11078: unify per-PR + nightly arm64 builds into build.yml May 27, 2026

@inikep inikep left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nogueiraanderson
nogueiraanderson merged commit 772c0b2 into 8.0 May 27, 2026
6 of 45 checks passed
@nogueiraanderson
nogueiraanderson deleted the PS-11078-arm64-nightly branch May 27, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants