ci: fix dry-run job graph, embed deploy commands in DEPLOY.md, correct runbook (#4040)

## Summary of Changes

* **Fix dry runs dead-ending after `stage-programs`** (found on runs
[29106365876](https://github.com/malbeclabs/doublezero/actions/runs/29106365876)
and
[29109225537](https://github.com/malbeclabs/doublezero/actions/runs/29109225537)):
the job-level `if: !inputs.dry_run` skip on `push-tags` transitively
skipped every downstream job with a default status condition — a skipped
ancestor poisons `success()` for the whole graph below it, even past
`verify-cloudsmith`'s own `if:` override — so `gate-programs` through
`announce` could never run in dry-run mode (confirmed: no deployment
record was ever created for `gate-programs`; it skipped in graph
evaluation before environment processing). The tag reusable workflow
gains a `dry_run` input that makes each tag job a validated no-op,
`push-tags` now runs in every mode, `verify-cloudsmith`'s special-case
condition is deleted, and the tag-approval Slack nudge fires in dry runs
too. Net effect: dry runs now traverse the full graph, and exercise the
same `testnet` tag-approval prompt as a real release (one extra approval
click per dry run).
* The generated `DEPLOY.md` embeds the exact program-deploy commands —
`cd` into the staged release dir, the three `solana program deploy`
invocations with keypairs under `~/testnet-ops/` (absolute: the [infra
runbook](https://github.com/malbeclabs/infra/blob/main/docs/runbooks/deploys/solana-programs-testnet.md)'s
relative `./testnet-ops/` paths assume `~`, no longer true once the
operator cds into `/opt/doublezero/program-releases/vX.Y.Z/`), and
`doublezero init` — replacing the retired Notion reference.
* Runbook: dry-run section documents the no-op tag jobs and full
approval sequence; recovery guidance prefers "Re-run all jobs" after
mid-pipeline failures and says to verify a re-run actually executed the
expected jobs before trusting its green check.

## Testing Verification

* The dry-run dead-end reproduced identically on a fresh dispatch (not
just re-runs), with `gate-programs` skipped the second its needs
completed, zero annotations, and no deployment record — ruling out
environment protection and matching transitive-skip semantics exactly.
* `DEPLOY.md` generation block extracted from the YAML and rendered in a
sandbox: multi-line commands, literal `~` paths, and interpolated
version all correct.
* actionlint on both workflows: only the pre-existing SC2086 (agave
install idiom, untouched).
* Full validation is the next dry run after merge (fresh dispatch —
workflow-file changes don't apply to re-runs).

Follow-up (not in this PR): a `pipeline-complete` guard job so a hollow
"success" fails loudly.

Author: ben-dz

.github/workflows/release.testnet.push.tags.yml

@@ -16,6 +16,11 @@ on:
         required: false
         type: boolean
         default: false
+      dry_run:
+        description: 'Validate inputs and succeed without pushing anything (for orchestrator dry runs)'
+        required: false
+        type: boolean
+        default: false
     secrets:
       DOUBLEZERO_PAT:
         description: 'PAT to push tags to the repository'
@@ -48,6 +53,11 @@ jobs:
           fi
           echo "Version format is valid."
 
+          if [ "${{ inputs.dry_run }}" = "true" ]; then
+            echo "Dry run — not pushing tag '$TAG_NAME'."
+            exit 0
+          fi
+
           if EXISTING=$(git rev-parse -q --verify "refs/tags/$TAG_NAME^{commit}"); then
             if [ "${{ inputs.skip_existing }}" = "true" ]; then
               HEAD_SHA=$(git rev-parse HEAD)

.github/workflows/release.testnet.yml

@@ -205,9 +205,7 @@ jobs:
           fi
       # The tag jobs request the `testnet` environment prompt again right after
       # this job completes; nudge the thread so that second approval isn't missed.
-      # Skipped in dry runs, where push-tags doesn't run at all.
       - name: Post tag-approval nudge to Slack thread
-        if: ${{ !inputs.dry_run }}
         uses: ./.github/actions/slack-thread-post
         with:
           token: ${{ secrets.SLACK_BOT_TOKEN }}
@@ -216,8 +214,11 @@ jobs:
           text: |-
             Version PRs confirmed merged. The 9 tag jobs will request the `testnet` environment prompt momentarily: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
 
+  # Runs in dry runs too (as a validated no-op via dry_run): a job-level skip
+  # here would transitively skip every downstream job with a default status
+  # condition — a skipped ancestor poisons `success()` for the whole graph
+  # below it, even past jobs that override their own condition.
   push-tags:
-    if: ${{ !inputs.dry_run }}
     needs: gate-tags
     uses: ./.github/workflows/release.testnet.push.tags.yml
     permissions:
@@ -238,14 +239,13 @@ jobs:
       version: v${{ inputs.version }}
       component: ${{ matrix.component }}
       skip_existing: true
+      dry_run: ${{ inputs.dry_run }}
     secrets:
       DOUBLEZERO_PAT: ${{ secrets.DOUBLEZERO_PAT }}
 
   verify-cloudsmith:
     runs-on: ubuntu-latest
     needs: [preflight, gate-tags, push-tags]
-    # In dry-run, push-tags is skipped; still verify (against the current version) to exercise the query logic.
-    if: ${{ !cancelled() && needs.gate-tags.result == 'success' && (needs.push-tags.result == 'success' || (inputs.dry_run && needs.push-tags.result == 'skipped')) }}
     timeout-minutes: 75
     steps:
       - name: Install Cloudsmith CLI
@@ -305,6 +305,7 @@ jobs:
           cp target/deploy/doublezero_serviceability.so staged/
           cp target/deploy/doublezero_telemetry.so staged/
           cp target/deploy/doublezero_geolocation.so staged/
+          # shellcheck disable=SC1003 # trailing backslashes inside single quotes are literal line-continuations for the generated markdown
           {
             echo "# Testnet program deploy v${VERSION}"
             echo
@@ -318,12 +319,33 @@ jobs:
             echo '```'
             echo
             echo "## Deploy"
-            echo "Follow the Notion runbook 'Build and deploy DoubleZero solana programs - testnet',"
-            echo "using these prebuilt artifacts instead of building locally."
-            echo "After deploying, set the serviceability program version:"
-            echo "  doublezero --env testnet global-config set-version --min-compatible-version <X.Y.Z>"
-            echo "(serviceability is the only program with a settable version account; telemetry has"
-            echo "none and geolocation has no CLI setter). Verify with: doublezero --env testnet version"
+            echo
+            echo "Runbook: https://github.com/malbeclabs/infra/blob/main/docs/runbooks/deploys/solana-programs-testnet.md"
+            echo "(its Build/scp section is superseded by these prebuilt, staged artifacts)"
+            echo
+            echo '```bash'
+            echo "cd /opt/doublezero/program-releases/v${VERSION}"
+            echo
+            echo 'solana program deploy \'
+            echo '  --program-id ~/testnet-ops/serviceability-program-keypair.json \'
+            echo '  -k ~/testnet-ops/serviceability-program-authority.json \'
+            echo '  doublezero_serviceability.so'
+            echo
+            echo 'solana program deploy \'
+            echo '  --program-id ~/testnet-ops/telemetry-program-keypair.json \'
+            echo '  -k ~/.config/doublezero/id.json \'
+            echo '  doublezero_telemetry.so'
+            echo
+            echo 'solana program deploy \'
+            echo '  --program-id ~/testnet-ops/geolocation-program-testnet-keypair.json \'
+            echo '  -k ~/.config/doublezero/id.json \'
+            echo '  doublezero_geolocation.so'
+            echo
+            echo '# refresh the onchain program version'
+            echo 'doublezero init'
+            echo '```'
+            echo
+            echo "Verify with: doublezero --env testnet version"
             echo
             echo "Then approve the waiting 'gate-programs' job (testnet environment) on the orchestrator run above."
           } > staged/DEPLOY.md

CHANGELOG.md

@@ -47,6 +47,8 @@ All notable changes to this project will be documented in this file.
   - Auto-publish the mainnet-beta client base image daily when the stable Cloudsmith channel advances (idempotency-gated so a run with no new version is a no-op), and notify `doublezero-edge-connect` to rebuild its testnet and mainnet-beta variants when a new base image is published. Serialize publishes with per-job concurrency groups, keep the notify steps non-fatal to the publish, and skip Debian pre-release versions when resolving the mainnet-beta tag. (#3990)
   - Add a testnet release orchestrator workflow (`release.testnet.yml`) that drives the release end to end: preflight checks, version-bump PRs for doublezero and infra, a human-approved gate before pushing the 9 component tags, CloudSmith package verification, Solana program build and staging with a manual deploy gate, onchain version verification, infra core/client deploys, QA, and Slack notifications. Supports `dry_run` for plumbing validation and safe re-runs (existing PRs are reused; already-pushed tags are skipped via a new `skip_existing` input on the tag workflow). Runbook at `docs/testnet-release.md`.
   - Thread all testnet release orchestrator Slack posts under a single per-run parent message in `#bots`, posted via the Slack Web API (`chat.postMessage` with a bot token) instead of incoming webhooks, which cannot start threads. Adds a threaded PR-links post after `open-prs` covering the merge-both-PRs / approve-gate-1 human steps, and a tag-approval nudge after `gate-tags` for the tag jobs' second `testnet` environment prompt. Slack failures degrade to workflow warnings and flat posts, never failing the release. (#4036)
+  - Testnet release: the generated `DEPLOY.md` now embeds the exact program-deploy commands (keypairs under `~/testnet-ops/`, artifacts from the staged release directory, `doublezero init` to refresh the onchain version) and links the infra deploy runbook, which replaced the Notion doc. Runbook recovery guidance updated to prefer "Re-run all jobs" after mid-pipeline failures.
+  - Fix dry runs dead-ending after `stage-programs`: the job-level skip of `push-tags` transitively skipped every downstream default-condition job (a skipped ancestor poisons `success()` for the whole graph, past `verify-cloudsmith`'s own override), so `gate-programs` through `announce` never ran in dry-run mode. The tag workflow gains a `dry_run` input and the tag jobs now run as validated no-ops instead of skipping — which also means dry runs exercise the `testnet` tag-approval prompt — and `verify-cloudsmith`'s special-case condition is deleted.
 
 ## [v0.29.0](https://github.com/malbeclabs/doublezero/compare/client/v0.28.0...client/v0.29.0) - 2026-07-02
 

docs/testnet-release.md

@@ -55,7 +55,7 @@ gh workflow run release.testnet.yml -R malbeclabs/doublezero -f version=X.Y.Z
 | `push-tags` | Pushes the 9 component tags (`controller`, `internet-latency-collector`, `agent`, `device-telemetry-agent`, `geoprobe-agent`, `geoprobe-target`, `funder`, `monitor`, `client`) via the reusable tag workflow, which runs in the protected `testnet` environment. | **Approve the `testnet` environment prompt** on the tag jobs (nudged in the Slack thread). |
 | `verify-cloudsmith` | Polls CloudSmith (up to ~60 min) until all 9 packages exist at the new version. | — |
 | `build-programs` | Builds the three Solana programs (`serviceability` default features; `telemetry` and `geolocation` with `--features testnet`) from main and uploads them with checksums and a `DEPLOY.md` manifest. | — |
-| `stage-programs` | Dispatches the infra `stage-programs.testnet.yml` workflow, which copies the artifacts to `nyc-tn-bm2:/opt/doublezero/program-releases/vX.Y.Z/`, then pings Slack. | **Approve infra's `testnet` environment** on the dispatched run (link posted to `#bots`). Then **deploy the programs** on nyc-tn-bm2 from that directory per the Notion runbook ("Build and deploy DoubleZero solana programs - testnet"), and set the onchain version. |
+| `stage-programs` | Dispatches the infra `stage-programs.testnet.yml` workflow, which copies the artifacts to `nyc-tn-bm2:/opt/doublezero/program-releases/vX.Y.Z/`, then pings Slack. | **Approve infra's `testnet` environment** on the dispatched run (link posted to `#bots`). Then **deploy the programs** on nyc-tn-bm2 following the `DEPLOY.md` staged alongside them (commands mirror the [infra runbook](https://github.com/malbeclabs/infra/blob/main/docs/runbooks/deploys/solana-programs-testnet.md)), and refresh the onchain version (`doublezero init`). |
 | `gate-programs` | Waits on the `testnet` environment. | **Approve gate 2** once the programs are deployed and the onchain version is set. |
 | `verify-onchain` | Installs the released client from CloudSmith and checks `doublezero --env testnet version` reports the new program version. | — |
 | `deploy-core` | Dispatches infra `deploy-core.testnet.yml` and waits for it. | **Approve infra's `testnet` environment** on the dispatched run (link posted to `#bots`). |
@@ -78,26 +78,41 @@ advancing the release.
 
 - Both version PRs are opened as **drafts** with `[DRY RUN]` titles. Do not merge them;
   gate 1 only checks that they were not closed.
-- No tags are pushed. `verify-cloudsmith` still runs, querying the **current** (previous)
-  version to exercise the CloudSmith query logic.
+- No tags are pushed: the tag jobs run as validated no-ops (`dry_run=true` on the tag
+  workflow). They run rather than skip because a job-level skip would transitively
+  skip every downstream job — a skipped ancestor poisons the default `success()`
+  condition for the whole graph below it. `verify-cloudsmith` still runs, querying
+  the **current** (previous) version to exercise the CloudSmith query logic.
 - Programs are still built and staged, but `verify-onchain` is skipped, and the infra
   deploy workflows are dispatched in their check mode (`mode=dry-run`).
-- The approval prompts are **not** skipped: gate 1, gate 2, the `testnet` environment
-  on this repo, and infra's `testnet` environment all still require approval even
-  though nothing is deployed.
+- The approval prompts are **not** skipped: gate 1, the `testnet` prompt on the
+  (no-op) tag jobs, gate 2, and infra's `testnet` environment all still require
+  approval even though nothing is deployed. A dry run exercises the same approval
+  sequence as a real release.
 
 Cleanup after a dry run: close both draft PRs and delete their branches
 (`release/vX.Y.Z` in doublezero, `release/testnet-vX.Y.Z` in infra).
 
 ## Recovery / re-running
 
-Use "Re-run failed jobs" on the orchestrator run to resume from where it stopped:
+After a mid-pipeline failure, prefer **"Re-run all jobs"** (or a fresh dispatch) over
+"Re-run failed jobs". Full re-runs are cheap and always correct here; partial re-runs
+depend on GitHub reviving the failure-skipped downstream jobs, and a hollow
+"success" that silently skips deploys/QA/announce is the failure mode to avoid
+(run 29106365876 ended exactly that way — root cause was a dry-run-only job-skip
+bug, since fixed, but the conservative habit stands). After any re-run, confirm the
+run actually executed the jobs you expected before trusting its green check.
+
+Full re-runs are safe by design:
 
 - The version PRs are reused if they already exist (the branch is force-pushed and the
   open PR is found by head branch).
 - Already-pushed tags are skipped (`skip_existing=true` on the tag workflow), so a
   partially completed tag matrix is safe to re-run.
-- Environment gates prompt for approval again on re-run.
+- Program staging re-copies the same artifacts; environment gates prompt again.
 
-If a downstream infra workflow failed, fix the cause there first; re-running the
-orchestrator job dispatches a fresh run of that workflow.
+If a dispatched infra workflow failed, fix the cause there first — the orchestrator
+dispatches those workflows fresh from infra `main` at runtime, so infra-side fixes
+apply on the next re-run without any doublezero change. Fixes to
+`release.testnet.yml` itself always need a fresh dispatch: any re-run (failed or all)
+executes the workflow snapshot from the original dispatch.