docs(skills): rewrite release-promotion — automated factory, no manual gates

castrojo · castrojo · commit 065688836fe0 · 2026-06-22T22:27:35.000-04:00
- use_merge_queue: true is unconditional and required, not event-conditional
- action_required on promotion PR checks is expected and normal (org blocks
  github-actions[bot] PR runs); merge_group events bypass this
- documents why merge queue is required: gh pr merge --auto is blocked by the
  merge queue ruleset, enqueuePullRequest is the only working path
- ruleset updated: main-merge-queue-no-review (id 18008292) replaces
  main-review-required-with-renovate-bypass (deleted)
- required_approving_review_count is 0 — promotion is fully automated
- adds DELETE+POST pattern for ruleset management (PATCH returns 404)
- removes all guidance saying use_merge_queue should be conditional
- removes 'preserves human approval gates' from verification (wrong frame)
- ci-reference.md: ruleset table updated, stale required_approving_review_count=1
  example corrected to 0

Assisted-by: Claude Sonnet 4.6 via GitHub Copilot
diff --git a/docs/skills/ci-reference.md b/docs/skills/ci-reference.md
@@ -235,15 +235,17 @@ PRs created by a workflow using `GITHUB_TOKEN` do NOT fire `pull_request` events
 
 ## Ruleset
 
-Ruleset: `main-review-required-with-renovate-bypass`
+Ruleset: `main-merge-queue-no-review` (id: 18008292)
 
 | Rule | Value |
 |---|---|
-| Required reviews | 1 approving review |
-| Required status checks | `validate` + `e2e` |
-| Merge queue | ALLGREEN, max_entries_to_build=1, check_response_timeout=120 min |
+| Required reviews | 0 (fully automated — no human approval) |
+| Required status checks | `validate` (strict) |
+| Merge queue | SQUASH, ALLGREEN, max_entries_to_build=2, timeout=120 min |
 | Bypass actors | OrganizationAdmin, Renovate, mergeraptor |
 
+**`action_required` on promotion PR checks is expected.** The org blocks `github-actions[bot]`-triggered `pull_request` workflow runs. `validate` and all other PR checks show `action_required` on every promotion PR — this is normal and not a failure. The merge queue fires `merge_group` events which bypass the bot approval policy and run `validate` cleanly.
+
 **e2e change detection:** `e2e` only tests PRs touching `elements/`, `files/`, `patches/`, `Justfile`, or `project.conf`. For all other paths (e.g. workflow pin bumps) the `e2e` job is skipped, which satisfies the required check. The `should-run` job uses `git diff` against the PR base — no `paths:` filter on the trigger.
 
 **Critical:** Required status checks must only include checks that fire on `pull_request`. A check that only fires on `merge_group` will permanently block the "Add to merge queue" button.
@@ -1818,8 +1820,8 @@ gh api repos/projectbluefin/dakota/branches/main/protection \
   "required_status_checks": {"strict": false, "checks": [{"context": "validate", "app_id": -1}]},
   "enforce_admins": false,
   "required_pull_request_reviews": {
-    "dismiss_stale_reviews": true, "require_code_owner_reviews": true,
-    "required_approving_review_count": 1
+    "dismiss_stale_reviews": false, "require_code_owner_reviews": false,
+    "required_approving_review_count": 0
   },
   "restrictions": null
 }
diff --git a/docs/skills/release-promotion.md b/docs/skills/release-promotion.md
@@ -1,6 +1,6 @@
 ---
 name: release-promotion
-description: Dakota publish and promotion flow from main to testing to stable, including promotion PRs, release gate behavior, and manual recovery. Use when working on promotion workflows, action_required gates, release execution, or stable-cut logic.
+description: Dakota publish and promotion flow from testing to main to stable, including promotion PRs, merge queue wiring, release gate behavior, and manual recovery. Use when working on promote-testing-to-main.yml, action_required on promotion PRs, merge queue setup, branch protection, or stable-cut logic.
 metadata:
   context7-sources:
     - /websites/github_en_actions
@@ -12,8 +12,10 @@ metadata:
 ## Overview
 
 Dakota has **two promotion layers**:
-1. `main` merge → publish → `:testing`
-2. `testing` → promotion PR → stable release execution
+1. `testing` → promotion PR → merge → `main` → publish → `:testing` / `:latest` / `:stable`
+2. Weekly human-gated stable cut via `production` Environment (2 approvals)
+
+Promotion from `testing` to `main` is **fully automated** — no human approval required at the PR level. The only human gate is the weekly `production` Environment for stable.
 
 Do not conflate "publish is healthy" with "stable promotion is healthy".
 
@@ -24,7 +26,9 @@ Use when the task mentions:
 - `pr-release-gate.yml`
 - `execute-release.yml`
 - promotion PRs from `auto/promote-testing-to-main`
-- `action_required` release gates
+- `action_required` on promotion PR checks
+- merge queue, `use_merge_queue`, `enqueuePullRequest`
+- branch protection or ruleset on `main`
 - stable release, `:latest`, `:stable`, or promotion PR flow
 
 ## When NOT to Use
@@ -40,19 +44,20 @@ Use when the task mentions:
 2. **Identify the stage.**
    - publish to `:testing`
    - open/update promotion PR
-   - gate the promotion PR
+   - gate the promotion PR (cosign verify)
    - execute stable release after merge
-3. **Check whether the gate is real or infrastructure.**
-   - `action_required` on a promotion PR often means the gate is waiting on policy or verification, not that the YAML crashed.
-4. **For promotion PR workflows, inspect reusable caller permissions first.**
+3. **`action_required` on promotion PR checks is expected and normal.**
+   The org blocks `github-actions[bot]`-triggered PR workflow runs. This means
+   `validate` and all other `pull_request` checks show `action_required` on every
+   promotion PR. This is NOT a failure — the merge queue bypasses it.
+4. **The merge queue is the only correct auto-merge path for dakota.**
+   `gh pr merge --auto` (`enablePullRequestAutoMerge`) is blocked by the merge queue
+   ruleset. Only `enqueuePullRequest` (triggered by `use_merge_queue: true`) works.
 5. **Do not add e2e back into the promotion PR path.**
    Dakota intentionally gates stable at the later human-approved release stage.
 6. **Automatic promotion cadence is Tuesday 04:00 UTC.**
    That schedule re-evaluates the promotion PR for the weekly stable cut.
-7. **Manual `workflow_dispatch` must preserve queue behavior.**
-   `use_merge_queue` should be true for `schedule` and `workflow_dispatch`, but
-   not for ordinary `push` refreshes from `testing`.
-8. **For manual recovery, re-run the failed publish/promote workflow that owns the stage, not some nearby check.**
+7. **For manual recovery, re-run the failed publish/promote workflow that owns the stage.**
 
 ## Promotion Map
 
@@ -64,20 +69,58 @@ push to testing (BST-affecting paths)
   → promote-testing-to-main.yml
       → auto/promote-testing-to-main PR
            → pr-release-gate.yml (cosign verify :testing)
-           → auto-merge → push to main
-               → execute-release.yml (commit message gate)
-                   → :latest / :stable + GitHub Release
+           → enqueuePullRequest → merge queue
+               → merge_group event → validate check (bypasses action_required)
+               → merge to main
+                   → execute-release.yml (commit message gate)
+                       → :latest / :stable + GitHub Release
 ```
 
+## Branch Protection and Ruleset State
+
+Ruleset: `main-merge-queue-no-review` (id: 18008292)
+
+| Rule | Value |
+|---|---|
+| Required reviews | 0 (fully automated) |
+| Required status checks | `validate` (strict) |
+| Merge queue | SQUASH, ALLGREEN, max_entries_to_build=2, timeout=120 min |
+| Bypass actors | OrganizationAdmin (always), Renovate (PR), mergeraptor (PR) |
+| Non-fast-forward | enforced |
+| Deletion | blocked |
+
+Classic branch protection on `main`: `required_approving_review_count: 0`.
+
+**Never re-add a required review count to classic protection or the ruleset.** It blocks every automated promotion PR permanently — the GHA bot cannot approve its own PRs.
+
+## Workflow Configuration
+
+`promote-testing-to-main.yml` must always have `use_merge_queue: true`:
+
+```yaml
+jobs:
+  promote:
+    uses: projectbluefin/actions/.github/workflows/reusable-promote-squash.yml@v1
+    with:
+      variants: '[{"image":"dakota"},{"image":"dakota-nvidia"}]'
+      cosign_identity_regexp: >-
+        ^https://github\.com/projectbluefin/(dakota|actions)/\.github/workflows/
+      run_e2e: false
+      use_merge_queue: true
+```
+
+Do not make `use_merge_queue` conditional on event type. It must always be `true`.
+
 ## Hard Rules
 
 - `promote-testing-to-main.yml` is a thin caller. Treat caller-level `permissions:` as critical.
 - `pr-release-gate.yml` must not starve the reusable gate token.
-- Promotion PRs do **not** run the full e2e quality gate; that belongs at the later stable promotion stage.
-- When a gate is `action_required`, inspect what policy condition it is waiting on before editing workflow code.
+- Promotion PRs do **not** run the full e2e quality gate; that belongs at the weekly stable gate.
+- `use_merge_queue: true` — unconditional, always. Not conditional on `github.event_name`.
+- The merge queue ruleset (`merge_queue` rule type) must exist on `main`. Without it, `enqueuePullRequest` fails silently.
+- `required_approving_review_count` must be 0 in both ruleset and classic branch protection.
 - Weekly automatic stable evaluation runs Tuesday at `0 4 * * 2`.
 - Keep `run_e2e: false` in dakota's promotion caller.
-- `use_merge_queue` should only enqueue on `schedule` or `workflow_dispatch`.
 
 ## Manual Recovery Shortcuts
 
@@ -91,75 +134,139 @@ gh run list --repo projectbluefin/dakota --workflow 'PR Release Gate' --limit 10
 
 # recent promote runs
 gh run list --repo projectbluefin/dakota --workflow 'Promote testing to main' --limit 10
+
+# verify ruleset is correct
+gh api repos/projectbluefin/dakota/rulesets | jq '[.[] | {id, name}]'
+gh api repos/projectbluefin/dakota/rulesets/18008292 | jq '[.rules[].type]'
+
+# verify classic branch protection has 0 reviews
+gh api repos/projectbluefin/dakota/branches/main/protection \
+  | jq '.required_pull_request_reviews.required_approving_review_count'
+```
+
+## Why `use_merge_queue: true` is required (not optional)
+
+The `projectbluefin` org blocks `github-actions[bot]`-triggered PR workflow runs.
+Every workflow fired by the promotion PR (`pull_request` event) gets `action_required`
+conclusion — the run is paused waiting for org approval. This means `validate` is never
+posted as a passing check, so `gh pr merge --auto` waits forever.
+
+The merge queue fires `merge_group` events instead of `pull_request` events. `merge_group`
+is NOT subject to the bot approval policy. `validate` runs clean, passes, and the queue
+merges. This is why `use_merge_queue: true` is unconditional — without it, the promotion
+PR never merges automatically.
+
+Bluefin uses exactly this pattern. Dakota must match it.
+
+## Ruleset Management
+
+The ruleset cannot be updated via `PATCH /repos/.../rulesets/{id}` with a standard
+`repo`-scoped PAT — returns 404. Use DELETE + POST to replace it:
+
+```bash
+# Delete old ruleset
+curl -X DELETE \
+  -H "Authorization: Bearer $(gh auth token)" \
+  -H "Accept: application/vnd.github+json" \
+  "https://api.github.com/repos/projectbluefin/dakota/rulesets/{OLD_ID}"
+
+# Create new ruleset
+curl -X POST \
+  -H "Authorization: Bearer $(gh auth token)" \
+  -H "Accept: application/vnd.github+json" \
+  -H "Content-Type: application/json" \
+  "https://api.github.com/repos/projectbluefin/dakota/rulesets" \
+  -d '{
+    "name": "main-merge-queue-no-review",
+    "target": "branch",
+    "enforcement": "active",
+    "conditions": {"ref_name": {"include": ["refs/heads/main"], "exclude": []}},
+    "bypass_actors": [
+      {"actor_id": null, "actor_type": "OrganizationAdmin", "bypass_mode": "always"},
+      {"actor_id": 2740, "actor_type": "Integration", "bypass_mode": "pull_request"},
+      {"actor_id": 3069633, "actor_type": "Integration", "bypass_mode": "pull_request"}
+    ],
+    "rules": [
+      {"type": "pull_request", "parameters": {
+        "required_approving_review_count": 0,
+        "dismiss_stale_reviews_on_push": true,
+        "require_code_owner_review": false,
+        "require_last_push_approval": false,
+        "required_review_thread_resolution": true,
+        "allowed_merge_methods": ["squash"]
+      }},
+      {"type": "required_status_checks", "parameters": {
+        "strict_required_status_checks_policy": true,
+        "do_not_enforce_on_create": false,
+        "required_status_checks": [{"context": "validate", "integration_id": 15368}]
+      }},
+      {"type": "merge_queue", "parameters": {
+        "merge_method": "SQUASH",
+        "max_entries_to_build": 2,
+        "min_entries_to_merge": 1,
+        "max_entries_to_merge": 5,
+        "min_entries_to_merge_wait_minutes": 5,
+        "grouping_strategy": "ALLGREEN",
+        "check_response_timeout_minutes": 120
+      }},
+      {"type": "non_fast_forward"},
+      {"type": "deletion"}
+    ]
+  }'
 ```
 
 ## Common Rationalizations
 
 | Rationalization | Reality |
 |---|---|
 | "Release gate is red, so publish is broken." | Different layer. Publish may be healthy while promotion is blocked. |
-| "Let's just add more checks to the promotion PR." | That slows humans and duplicates the real stable gate. |
-| "`action_required` means rerun the same gate." | Usually it means read the gate condition first. |
+| "Let's just add more checks to the promotion PR." | That slows the queue and duplicates the real stable gate. |
+| "`action_required` means rerun the same gate." | It means the org is blocking bot PR runs. Only the merge queue bypasses this. |
+| "`use_merge_queue` only needs to be true for schedule/dispatch." | Wrong. It must always be true. The bot approval block applies to push events too. |
+| "Adding a required review adds safety." | It permanently blocks every automated promotion PR. |
 | "This reusable caller only needs job-level permissions." | Wrong often enough to deserve a scar. Check top-level caller permissions first. |
 
 ## Red Flags
 
+- `use_merge_queue` is conditional on `github.event_name`
+- `required_approving_review_count` is greater than 0 in ruleset or classic protection
+- The ruleset is missing the `merge_queue` rule type
 - editing stable-promotion logic while the real failure is earlier publish plumbing
 - adding full e2e to the promotion PR path
-- rerunning arbitrary nearby workflows instead of the owning stage
-- ignoring `auto/promote-testing-to-main` PR state and debugging the wrong branch
+- rerunning `action_required` workflow runs — they cannot be rerun, they are bot-blocked
 
 ## Verification
 
-- [ ] You identified the exact promotion stage that is failing
-- [ ] You checked open promotion PR state before editing YAML
-- [ ] Reusable caller permissions were validated if the gate did not start
-- [ ] You did not collapse publish, promotion, and stable release into one mental model
-- [ ] Any change preserves the factory's intended human approval gates
+- [ ] `use_merge_queue: true` unconditionally in `promote-testing-to-main.yml`
+- [ ] Ruleset `main-merge-queue-no-review` exists and has `merge_queue` rule
+- [ ] Classic branch protection `required_approving_review_count` is 0
+- [ ] `run_e2e: false` unchanged in the caller
+- [ ] Promotion PR enqueued and merged without any human interaction
 - [ ] Weekly cadence remains Tuesday `04:00 UTC`
-- [ ] `workflow_dispatch` still supports the same enqueue path as the scheduled run
-- [ ] `run_e2e: false` stayed unchanged in the caller
+- [ ] You did not collapse publish, promotion, and stable release into one mental model
 
 ## Lessons Learned
 
-### release/blocked after CI-only push to testing is expected (2026-06-22)
+### release/blocked after CI-only push to testing is expected
 
 When a paths-ignored push (e.g. `.github/workflows/**` change) advances the `testing`
 HEAD, the promote gate runs against the new SHA and finds no CI results for it.
 The gate correctly sets `release/blocked` — the SHA has never been built.
 
-**This is not a pipeline failure.** Do not re-trigger or debug. The resolution is:
-
-1. The BST build for the prior SHA (triggered by the image-affecting push) completes.
+This is not a pipeline failure. The resolution is automatic:
+1. The BST build for the prior SHA completes.
 2. `publish.yml` fires → `:testing` updated.
-3. Tuesday 04:00 UTC schedule re-runs `promote-testing-to-main` → gate passes → auto-merge fires.
+3. Next promote run re-evaluates → gate passes → merge queue fires.
 
-**Factory pattern:** Enable auto-merge on the promotion PR immediately, even while
-`release/blocked`. The merge queue waits for the gate to clear — no human follow-up needed.
+### One BST build at a time — cancel everything before starting a new build
 
-```bash
-gh pr merge 970 --repo projectbluefin/dakota --squash --auto
-```
-
-### One BST build at a time — cancel everything before starting a new build (2026-06-22)
-
-**One BST build at a time.** Before triggering or landing any change that starts a new
-BST build, cancel ALL other in-progress BST jobs — including cache-warm runs.
-
-Concurrent BST builds compete for the same `ubuntu-24.04` runners and the same remote
-CAS write bandwidth. Running two builds simultaneously does not halve the time — it
-more than doubles it, and cache-warm runs will hit the 6h job timeout with
-`Cached elements after warm: 0`, wasting the entire run.
+Before triggering or landing any change that starts a new BST build, cancel ALL
+in-progress BST jobs — including cache-warm runs.
 
 ```bash
-# List all in-progress runs (cache-warm, stale builds, anything)
 gh run list --repo projectbluefin/dakota --json databaseId,status,name \
   | python3 -c "import json,sys; [print(r['databaseId'], r['name']) for r in json.load(sys.stdin) if r['status'] in ('in_progress','queued','pending')]"
-
-# Cancel each one before proceeding
 gh run cancel <run-id> --repo projectbluefin/dakota
 ```
 
-**Cache-warm runs are NOT exempt.** "Its progress is additive" is the rationalization
-that causes the failure. A cache-warm run competing with a real build starves both.
-Cancel the warm run, let the real build finish, then re-trigger cache-warm separately.
+Cache-warm runs are NOT exempt. Cancel everything, let one build finish, then re-trigger warm separately.
