qa-gate #648
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # qa-gate — the automated inverse of each plugin's own CI. | |
| # | |
| # Each first-party plugin repo runs its OWN lightweight CI on every push, proving "this plugin | |
| # works" against a real backend where one applies (see e.g. GetBusbar/store-postgres's real | |
| # postgres: service container). This workflow is the other direction: on every push to `qa` | |
| # (the pre-release integration branch — see the dev→qa→main branch-model note in | |
| # scripts/release-check.sh's header), it runs the real end-to-end release gate, proving "busbar | |
| # works against ALL its plugins together" at this exact commit. Neither direction substitutes for | |
| # the other. | |
| # | |
| # BRANCH MODEL: `dev` gets only the cheap per-push CI (ci.yml) — push there often. Promoting | |
| # dev→`qa` is what spends this real-money full-plugin gate; it is the pre-release soak. A green qa | |
| # is what earns a promotion qa→`main`, where tag-on-main.yml auto-cuts the release. This | |
| # intentionally does NOT run on `dev`, `main`, or PRs — `scripts/release-check.sh` documents itself | |
| # as a pre-release gate, not a per-commit one. `qa` is where that cost belongs: proving the promoted | |
| # commit is release-ready before it ever reaches `main`. | |
| # | |
| # ── THIS FILE IS A DISPATCHER, NOT THE GATE ────────────────────────────────────────────────────── | |
| # | |
| # `workflow_run` ALWAYS loads the workflow file from the DEFAULT branch. Whatever this file looks | |
| # like on `main` is what auto-fires, regardless of what is on `qa`. Measured on qa c736177: the | |
| # auto-fired gate ran ONE job while the whole segmentation umbrella sat unused on `qa`. So a gate | |
| # improvement could not gate the release that shipped it, and it failed SILENTLY: the run went | |
| # green, it had just done less than anyone thought. | |
| # | |
| # Hence: this file checks out the TRIGGERING SHA and invokes `scripts/qa-gate-run.sh` FROM THAT | |
| # CHECKOUT. Gate logic rides the commit it gates; change the script on `qa` and the next auto-fired | |
| # gate runs the new logic with no `main` edit and no dormancy window. What stays here is only what | |
| # GitHub must read before a checkout exists, or what is structural to the run graph: `on:`, | |
| # `concurrency`, permissions, env/secrets, `runs-on`, `timeout-minutes`, the `needs`/`if` graph, | |
| # and the `strategy.matrix` EXPRESSION (its CONTENTS come from the script, as a job output). | |
| # | |
| # ── THE SHAPE OF THE RUN, AND WHY ──────────────────────────────────────────────────────────────── | |
| # | |
| # build ─┐ compile ONCE, upload target/ as an artifact | |
| # fast ──┼─→ slow (matrix) ─┐ one leg per live-mock segment, HYDRATES, never rebuilds | |
| # └─→ loader ──┼─→ umbrella (if: !cancelled()) | |
| # ┘ | |
| # | |
| # On a green run wall clock is max(job), not sum(job), so matrix WIDTH and BALANCE are the entire | |
| # win, and per-job FIXED COST is the entire risk, because it is paid N times. This workspace builds | |
| # with `lto = "fat"` + `codegen-units = 1`; ten legs each paying that link would make wall clock | |
| # WORSE than the old sequential gate while looking like progress. `build` pays it once and every leg | |
| # hydrates from the artifact. See qa-gate-run.sh's `hydrate` for the mtime trap that makes a naive | |
| # artifact restore a silent no-op, and for why Swatinem/rust-cache alone cannot do this job. | |
| # | |
| # `fast` does NOT gate `build`; they are siblings. Serialising a cheap tier in FRONT of the long | |
| # pole would spend 100% of its duration on every green run and save nothing. But `slow` needs BOTH, | |
| # which costs zero wall clock for as long as fast < build, while still keeping the real-money legs | |
| # from starting behind a broken fast tier. That inequality is load-bearing, so `fast` carries a | |
| # deliberately tight timeout-minutes: if it ever grows into the pole, it fails instead of quietly | |
| # eating the budget. | |
| # | |
| # THE CEILING ON ALL OF THIS is the account-wide concurrent-job cap. If the matrix is wider than the | |
| # runners available, legs queue and wall clock silently becomes sum-of-batches again. That is why | |
| # reserved segments get no runner of their own (qa-gate-run.sh reports them inside `fast`) and why | |
| # legs are spent only on work that actually runs. | |
| # | |
| # ── WHAT THIS DOES *NOT* YET BUY, STATED PLAINLY ──────────────────────────────────────────────── | |
| # | |
| # This change removes ONE pole: the N-times rebuild of the busbar workspace. Two others are still | |
| # standing, and neither is fixable from this file: | |
| # | |
| # 1. release-check.sh does not partition yet. `--segment <id>` is accepted and LABELLED, but the | |
| # script still runs its FULL coverage-preserving gate for every label (its own header says so, | |
| # with a TODO). So today a fan-out of N segments runs the whole ~2h gate N times in parallel: | |
| # wall clock is unchanged, and cost goes up. The fan-out only converts into wall clock once | |
| # that per-segment phase partitioning lands. This workflow is built to be READY for it, not to | |
| # deliver it alone. | |
| # 2. The plugin SIBLING workspaces are each built cold, per leg. They are separate workspaces with | |
| # separate target dirs, so the build-once artifact cannot help them, and nothing here caches | |
| # them. Once (1) lands, `cargo build/test --workspace --release` inside each sibling becomes | |
| # the new long pole. The lever is a per-sibling dependency cache in the `slow` leg, keyed by | |
| # the checkout dir — which means the matrix needs to know that dir. qa-gate-run.sh's `matrix` | |
| # already forwards any extra qa/segments.toml columns through, so a `checkout_dir` field on | |
| # the manifest is enough to wire it without reworking anything here. | |
| # | |
| # So: do not read a green run of this workflow as "the gate now takes 15 minutes". It takes | |
| # max(job), and until (1) lands max(job) is still a full release-check.sh. | |
| name: qa-gate | |
| on: | |
| # Runs AFTER ci.yml succeeds, not on every push directly -- this is a real, real-money e2e run; no | |
| # point spending it on a commit that hasn't even cleared the cheaper gate yet. Only fires when the | |
| # CI run that just finished was itself triggered by a push to `qa` (see the `if:` on the jobs | |
| # below), and only on conclusion == success. | |
| workflow_run: | |
| workflows: ["CI"] | |
| types: [completed] | |
| workflow_dispatch: {} | |
| # NO `permissions:` block, deliberately. The sibling-clone steps authenticate to OTHER GetBusbar | |
| # repos with `github.token`, and what that token can reach depends on the repo/org default. Pinning | |
| # it here would be a silent scope change riding along in a wall-clock rewrite, and the failure mode | |
| # is a warn-and-continue clone that quietly removes a plugin from the gate. Leave the auth surface | |
| # exactly as the gate that ran before this change had it. | |
| # release-check.sh documents up to ~2h; give real headroom before GitHub's own job timeout bites. | |
| # | |
| # Key the group on the TRIGGERING branch, NOT github.ref. For a workflow_run event github.ref is | |
| # ALWAYS refs/heads/<default-branch> (main), so every qa-gate run — including the ones a dev/main CI | |
| # completion spawns that then skip via the job `if:` — would share one group. With | |
| # cancel-in-progress, a skipped sibling (e.g. dev's CI finishing ~seconds after qa's) would CANCEL | |
| # the real in-flight qa gate. head_branch scopes the group to the branch whose CI actually fired, so | |
| # a qa-triggered run only ever supersedes another qa-triggered run; dev/main-triggered skips sit in | |
| # their own harmless groups. (workflow_dispatch has no workflow_run event → falls through to ref.) | |
| concurrency: | |
| group: qa-gate-${{ github.event.workflow_run.head_branch || github.ref }} | |
| cancel-in-progress: true | |
| env: | |
| BUILD_ARTIFACT: qa-gate-target-${{ github.run_id }} | |
| TARBALL: /tmp/busbar-target.tzst | |
| jobs: | |
| # ── fast: cheap tier + the matrix that drives the fan-out. Sibling of `build`, never in front of | |
| # it. Also the job that converts qa/segments.toml into strategy.matrix contents, because it has | |
| # the triggering SHA checked out and finishes long before anything needs the answer. | |
| fast: | |
| name: fast tier + matrix | |
| # Only run for a REAL CI success on `qa` (never on a failed/cancelled CI run, and never for CI | |
| # runs from other branches/PRs) -- workflow_dispatch always runs regardless. Downstream jobs | |
| # inherit this decision through `needs` (a skipped dependency skips its dependents), so the | |
| # condition is stated on the two root jobs only. | |
| if: >- | |
| github.event_name == 'workflow_dispatch' || | |
| (github.event.workflow_run.conclusion == 'success' && | |
| contains(fromJSON('["qa"]'), github.event.workflow_run.head_branch)) | |
| runs-on: ubuntu-latest | |
| # CORRECTED after the first real run. This job was originally a SIBLING of `build`, on the | |
| # reasoning that a 5s fast tier alongside a 157s build costs nothing. Both halves of that were | |
| # wrong in CI: the 5s was measured locally against a WARM target, and with no artifact this job | |
| # rebuilt the workspace from scratch (`export` alone took 153s). It also could not have passed, | |
| # because `hook-bindings` needs a cdylib only `build` produces. | |
| # It now hydrates from `build` like every other consumer, which removes a duplicate full build | |
| # rather than adding a wait. | |
| needs: [build] | |
| # Deliberately tight. This job sits on the critical path ahead of `slow`, so if it ever grows | |
| # into a long pole that must be a loud failure, not a silent regression against the budget. | |
| timeout-minutes: 15 | |
| outputs: | |
| matrix: ${{ steps.matrix.outputs.matrix }} | |
| steps: | |
| - uses: actions/checkout@v7 | |
| with: | |
| path: busbarAI | |
| # workflow_run doesn't check out the triggering commit by default (it defaults to the | |
| # default branch) -- pin explicitly to the exact SHA that CI just passed. Falls through | |
| # to the normal ref resolution on workflow_dispatch (no workflow_run event present). | |
| ref: ${{ github.event.workflow_run.head_sha || github.ref }} | |
| # REGISTRY-DRIVEN MATRIX, NO HAND-WRITTEN LEGS. qa/segments.toml is the source of truth; | |
| # qa-segments.sh --list already emits it as TSV expressly so CI can build its matrix. This | |
| # step turns that into JSON on a job output. Repartitioning the manifest (splitting `plugins` | |
| # into one segment per plugin repo, arming a reserved slot, adding a segment) therefore needs | |
| # ZERO edits here: this workflow only ever names `needs.fast.outputs.matrix`. | |
| - id: matrix | |
| name: Build the fan-out matrix from qa/segments.toml | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh matrix | |
| # HYDRATE, same as `slow`. The fast tier is NOT build-free: `hook-bindings` needs the | |
| # hook-test plugin cdylib, and `test_support` PANICS under CI when it is missing rather than | |
| # silently skipping the hook-plugin coverage (test_support/mod.rs, "refusing to silently | |
| # skip"). The first real run proved both halves of why this step has to be here: | |
| # * 39 hook tests failed on the missing cdylib, and | |
| # * `export` took 153s because with no artifact the fast job REBUILT the workspace from | |
| # scratch -- the exact duplicate build that build-once exists to remove. | |
| # So this is not a wall-clock cost: it REMOVES a redundant full build from the critical path | |
| # and is what makes this job genuinely fast. | |
| - uses: dtolnay/rust-toolchain@stable | |
| - uses: actions/download-artifact@v4 | |
| with: | |
| name: ${{ env.BUILD_ARTIFACT }} | |
| path: /tmp | |
| - name: Hydrate target/ from the build-once artifact (and assert it was accepted as fresh) | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh hydrate "$TARBALL" | |
| - name: Fast-tier segments + reserved-slot report | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh fast | |
| # ── build: the build-once stage. The single most important job in this graph (see the fixed-cost | |
| # note in the header). Everything downstream hydrates its target/ from this artifact. | |
| build: | |
| name: build once (shared by every leg) | |
| if: >- | |
| github.event_name == 'workflow_dispatch' || | |
| (github.event.workflow_run.conclusion == 'success' && | |
| contains(fromJSON('["qa"]'), github.event.workflow_run.head_branch)) | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 60 | |
| steps: | |
| - uses: actions/checkout@v7 | |
| with: | |
| path: busbarAI | |
| ref: ${{ github.event.workflow_run.head_sha || github.ref }} | |
| - uses: dtolnay/rust-toolchain@stable | |
| # rust-cache still earns its place HERE: it makes this one build faster run-over-run by | |
| # restoring the dependency graph. It cannot replace the artifact below, because it | |
| # deliberately drops workspace-member artifacts, and `busbar` IS where fat LTO is paid. | |
| - uses: Swatinem/rust-cache@v2 | |
| with: | |
| workspaces: busbarAI | |
| - name: Build once + prune + pack | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh build "$TARBALL" | |
| - uses: actions/upload-artifact@v4 | |
| with: | |
| name: ${{ env.BUILD_ARTIFACT }} | |
| path: ${{ env.TARBALL }} | |
| # This tarball exists only to be handed to the sibling jobs in THIS run. | |
| retention-days: 1 | |
| compression-level: 0 # already zstd-compressed; re-zipping it just burns critical path | |
| # ── slow: the live-mock fan-out. One independently-reported leg per active live-mock segment. | |
| # Matrix CONTENTS come entirely from qa/segments.toml via the `fast` job's output. | |
| slow: | |
| name: ${{ matrix.segment.id }} | |
| needs: [build, fast] | |
| runs-on: ubuntu-latest | |
| # Still generous, on purpose. release-check.sh today runs its FULL coverage-preserving gate for | |
| # any --segment label; per-segment PHASE PARTITIONING is the documented additive refinement that | |
| # actually converts this fan-out into wall clock. Until that lands, a leg can still be long, and | |
| # a too-tight timeout here would turn a known-slow gate red rather than slow. | |
| timeout-minutes: 180 | |
| strategy: | |
| # NO cross-segment masking: every segment runs and reports independently, which is the | |
| # umbrella's stated design property (qa/segments.toml, "green-on-completion"). One red plugin | |
| # must not hide the state of the other nine. | |
| fail-fast: false | |
| matrix: ${{ fromJSON(needs.fast.outputs.matrix) }} | |
| steps: | |
| - uses: actions/checkout@v7 | |
| with: | |
| path: busbarAI | |
| ref: ${{ github.event.workflow_run.head_sha || github.ref }} | |
| - uses: dtolnay/rust-toolchain@stable | |
| - uses: actions/download-artifact@v4 | |
| with: | |
| name: ${{ env.BUILD_ARTIFACT }} | |
| path: /tmp | |
| - name: Hydrate target/ from the build-once artifact (and assert it was accepted as fresh) | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh hydrate "$TARBALL" | |
| - name: Sibling checkouts (registry-driven from plugins.yaml) | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh siblings | |
| - name: Confirm Docker is available (Postgres/Valkey/MySQL/Vault phases need it) | |
| working-directory: busbarAI | |
| run: docker ps | |
| - name: Run segment ${{ matrix.segment.id }} | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh segment "${{ matrix.segment.id }}" | |
| # ── loader: coverage that lives in NEITHER release-check.sh NOR the segment manifest. These steps | |
| # were inline in the old single release-check job; folding that job into the fan-out would have | |
| # silently dropped them. Coverage is the constraint, wall clock only the objective, so they keep | |
| # their own job, and cheap, because it hydrates the same build-once artifact. | |
| loader: | |
| name: plugin-loader mechanism tests | |
| needs: [build, fast] | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 45 | |
| steps: | |
| - uses: actions/checkout@v7 | |
| with: | |
| path: busbarAI | |
| ref: ${{ github.event.workflow_run.head_sha || github.ref }} | |
| - uses: dtolnay/rust-toolchain@stable | |
| - uses: actions/download-artifact@v4 | |
| with: | |
| name: ${{ env.BUILD_ARTIFACT }} | |
| path: /tmp | |
| - name: Hydrate target/ from the build-once artifact (and assert it was accepted as fresh) | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh hydrate "$TARBALL" | |
| - name: Sibling checkouts (registry-driven from plugins.yaml) | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh siblings | |
| - name: Loader-mechanism tests against the real sibling-built store-sqlite-plugin | |
| working-directory: busbarAI | |
| run: ./scripts/qa-gate-run.sh loader | |
| # ── umbrella: the single required status. It must report even when a leg FAILED, so it cannot use | |
| # the default `if` (which requires every `needs` to have succeeded). `!cancelled()` rather than | |
| # `always()`: with cancel-in-progress a superseded run would otherwise end in a RED umbrella | |
| # instead of an honest `cancelled`, and a red gate nobody should act on is its own kind of noise. | |
| # It also has to RE-STATE the trigger condition, because `!cancelled()` breaks the `needs`-skip | |
| # cascade: without it this job would run, and pass over a graph of skipped jobs, on every | |
| # dev/main CI completion that the root jobs correctly skip. | |
| umbrella: | |
| name: qa-gate umbrella | |
| needs: [build, fast, slow, loader] | |
| if: >- | |
| !cancelled() && | |
| (github.event_name == 'workflow_dispatch' || | |
| (github.event.workflow_run.conclusion == 'success' && | |
| contains(fromJSON('["qa"]'), github.event.workflow_run.head_branch))) | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 5 | |
| steps: | |
| # `needs.slow.result` collapses the WHOLE matrix to one value, so this is red if any leg is. | |
| # 'skipped' is treated as red on purpose: with fail-fast: false the only way a leg skips is a | |
| # dependency failure or a cancel, and a gate that reports green over a tier that never ran is | |
| # exactly the silent-dormancy failure this rewrite exists to kill. | |
| - name: Assert every tier reported green | |
| run: | | |
| set -u | |
| fails=0 | |
| check() { | |
| printf ' %-10s %s\n' "$1" "$2" | |
| [ "$2" = "success" ] || fails=$((fails+1)) | |
| } | |
| echo "qa-gate tier results:" | |
| check build "${{ needs.build.result }}" | |
| check fast "${{ needs.fast.result }}" | |
| check slow "${{ needs.slow.result }}" | |
| check loader "${{ needs.loader.result }}" | |
| if [ "$fails" -ne 0 ]; then | |
| echo "qa-gate umbrella: RED ($fails tier(s) not green). DO NOT PROMOTE qa→main" >&2 | |
| exit 1 | |
| fi | |
| echo "qa-gate umbrella: GREEN" |