chore(ops): automating dev cluster

[SimonRastikian]

Closes partly #3934
Namely the dev cluster upgrade

[claude]

Pull request overview

This PR adds an interactive operator toolkit under scripts/ops/ for upgrading the NEAR One dev clusters, partially closing #3934. A generic helper library (common.sh) plus a dev-cluster-specific one (dev-common.sh) back three new scripts: a Nomad job image swap (migrate-dev-cluster.sh), a contract propose/vote flow (upgrade-dev-contract.sh), and two menus that sequence them in runbook order (nodes → verify → contract). It also renames prepare-release.sh to prepare-github-release.sh and updates the references in RELEASES.md and the PR-review prompt.

Changes:

• New shared helpers: die/require_cmds/check_version/confirm plus command echoing (fmt_cmd, show_cmd, show_output, run_cmd).
• New dev-cluster helpers: resolve_dev_cluster (network → contract, member accounts, deposits, and per-cluster endpoint env vars), verify_nodes (scrapes mpc_node_build_info from /metrics), test_sign (on-chain smoke signature).
• migrate-dev-cluster.sh: lists mpc-node* Nomad jobs, rewrites nearone/mpc-node-gcp:<tag> in-place via the HTTP API, plans, confirms, registers, and waits for the new allocation.
• upgrade-dev-contract.sh: fetches (release tarball or local build) the contract WASM, hand-rolls the borsh encoding of ProposeUpdateArgs { code: Some(..), config: None }, then drives propose_update + vote_update per member account.
• menu.sh / dev-menu.sh entry points; prepare-release.sh → prepare-github-release.sh rename with doc updates.

The borsh framing (0x01 + u32-LE length + bytes + 0x00) matches ProposeUpdateArgs field order in crates/near-mpc-contract-interface/src/types/updates.rs:59, the release tag/asset names match .github/workflows/release.yml:170, and the release= label matches crates/node/src/metrics.rs:402 — all verified. The .github/prompts/pr-review.prompt.md edit is a mechanical link rename only; no review-instruction changes.

Reviewed changes

Per-file summary

File	Description
.github/prompts/pr-review.prompt.md	Updates the two prepare-release.sh references to the renamed script.
RELEASES.md	Same rename applied throughout the release runbook.
scripts/ops/prepare-release.sh → scripts/ops/prepare-github-release.sh	Renamed; header and usage examples updated.
scripts/ops/common.sh	New generic sourced helpers (errors, dependency checks, semver check, confirm, command echoing).
scripts/ops/menu.sh	New top-level interactive menu dispatching to the release script or the dev-cluster menu.
scripts/ops/dev-cluster/dev-common.sh	New dev-cluster helpers: cluster resolution, node version verification, test sign.
scripts/ops/dev-cluster/dev-menu.sh	New orchestrator: network/version prompts, then nodes → verify → contract.
scripts/ops/dev-cluster/migrate-dev-cluster.sh	New Nomad image-swap step with plan/confirm/apply and allocation wait.
scripts/ops/dev-cluster/upgrade-dev-contract.sh	New contract step: fetch/build WASM, borsh-encode, propose_update, vote_update.

Findings

Blocking (must fix before merge):

• scripts/ops/dev-cluster/upgrade-dev-contract.sh:66 — the local-build path can never succeed. cargo near build emits target/near/<crate>/<crate>.wasm, i.e. target/near/mpc_contract/mpc_contract.wasm (see Makefile.toml:188, Makefile.toml:273, .github/workflows/ci.yml:294, .github/workflows/build_contract.yml:42, scripts/launch-localnet.sh:12), which is at depth 2. With -maxdepth 1 the find matches nothing, so MPC_WASM_SOURCE=build (and the (b)uild prompt) always dies at line 68 right after paying for a full contract build.

  Simply widening to -maxdepth 2 would trade one bug for a worse one: target/near also holds tee_verifier/tee_verifier.wasm and test_parallel_contract/test_parallel_contract.wasm, -newermt '-10 minutes' matches whichever were built recently, and find | head -1 has no defined ordering — so the script could serialize the wrong artifact and propose it as the contract update. Use the deterministic path instead (this also drops the GNU-only -newermt):

  built="${root}/target/near/mpc_contract/mpc_contract.wasm"
  [[ -f "$built" ]] || die "Expected ${built} after the cargo-near build."
  cp "$built" "$wasm"

• .cliffignore:2 — still points at the old path: # invoked from scripts/ops/prepare-release.sh and the manual flow in RELEASES.md. The rename updated RELEASES.md and the prompt file but missed this one; per CLAUDE.md §Documentation alignment, doc drift from a rename is a same-PR fix. Change to scripts/ops/prepare-github-release.sh.

Non-blocking (nits, follow-ups, suggestions):

• scripts/ops/dev-cluster/migrate-dev-cluster.sh:25 — NOMAD_TOKEN is passed as -H "X-Nomad-Token: ..." inside args, which is expanded onto the curl command line and is therefore visible in ps//proc/<pid>/cmdline. That directly contradicts the comment three lines below (:32, "-K - keeps the credentials out of the process list"), which only covers the basic-auth pair. Either move the header into the same -K - config stream (header = "X-Nomad-Token: ...") or narrow the comment so it doesn't overstate the protection.
• scripts/ops/dev-cluster/migrate-dev-cluster.sh:24 — curl -sf throws away the response body on an HTTP error, so die "Plan failed for ${job_id}." (:99) and die "Job registration failed for ${job_id}." (:107) surface no Nomad diagnostics at all — exactly when the operator needs them. --fail-with-body (curl ≥ 7.76) keeps the message.
• scripts/ops/dev-cluster/migrate-dev-cluster.sh:62 — a single transient failure of the allocation poll propagates through pipefail and set -e, killing the script mid-rollout and abandoning every remaining mpc-node-* job. Since this is a poll with a bounded retry budget, status=$(...) || status=unknown would let it ride out a blip.
• scripts/ops/dev-cluster/upgrade-dev-contract.sh:78 — sha256sum is GNU-only (macOS has shasum -a 256); combined with -newermt at :66 this makes the dev-cluster tooling Linux-only, whereas fix(scripts): make prepare-release.sh work on BSD/macOS #3273 deliberately made prepare-release.sh work on BSD/macOS. Worth keeping consistent, or documenting the restriction in the header.
• scripts/ops/dev-cluster/dev-common.sh:24 — "15 NEAR is just under the ~15.13 storage requirement" states a figure that isn't derivable and doesn't currently hold: the requirement is (PROPOSE_UPDATE_ENTRY_OVERHEAD_BYTES + wasm_bytes) * storage_byte_cost (crates/near-mpc-contract-interface/src/deposits.rs:14), which at the 1_235_000-byte cap in scripts/check-contract-wasm-size.sh is ≈12.4 NEAR. It's the stale-prone magic-number comment CLAUDE.md §Code Comments warns about — either point at ProposedUpdates::required_deposit or drop it (the excess is refunded at crates/contract/src/lib.rs:1496, so the constant is safe regardless).
• scripts/ops/dev-cluster/dev-common.sh:59 — test_sign is the only call in the PR using near-cli's deprecated JS-style syntax (near call … --accountId … --gas … --deposit …); everything else uses near-cli-rs (near contract call-function …). As a consequence it also ignores MPC_SIGN_WITH, so the sign-with-legacy-keychain escape hatch documented at upgrade-dev-contract.sh:10 doesn't apply to the smoke test.
• scripts/ops/common.sh:24 — check_version rejects pre-release versions, but .github/workflows/release.yml:39 accepts X.Y.Z-<suffix>. Release candidates therefore can't be rolled out with this tooling; loosening the regex to match the workflow would avoid a surprise mid-runbook.
• scripts/ops/dev-cluster/dev-menu.sh:70 — || true swallows a failed contract upgrade and the run still finishes with "Done." at :76. An explicit "contract step failed" line would keep the summary honest.
• No shellcheck job exists in CI (checked .github/, Makefile.toml, flake.nix), so the # shellcheck source= directives added here aren't enforced by anything. Adding a lint step over scripts/ops/** would be a cheap way to keep this tooling from rotting.
• RELEASES.md documents prepare-github-release.sh but nothing points at the new scripts/ops/menu.sh entry point or the dev-cluster upgrade flow. A short section would make the tooling discoverable to anyone who wasn't in this PR.

⚠️ Issues found

[SimonRastikian]

@claude review this PR

[SimonRastikian]

@claude review

[claude]

Pull request overview

This revision addresses the two previously-blocking items (the find -maxdepth 1 local-build path, now a deterministic target/near/mpc_contract/mpc_contract.wasm lookup, and the stale .cliffignore reference) plus most of the earlier nits: --fail-with-body replaces -sf on the Nomad calls, NOMAD_TOKEN moved into the -K - config stream, the allocation poll no longer aborts the run on a blip, sha256_of handles macOS, check_version accepts pre-release suffixes, test_sign uses near-cli-rs, and dev-menu.sh reports a failed contract step honestly. RELEASES.md also gained an "Ops tooling" section.

I re-verified the on-chain plumbing against the repo: the borsh framing (0x01 + u32-LE len + bytes + 0x00) matches ProposeUpdateArgs { code: Option<Vec<u8>>, config: Option<Config> } (crates/near-mpc-contract-interface/src/types/updates.rs:59); propose does not auto-vote (crates/contract/src/update.rs:163), so voting with the proposer is correct; the excess deposit is refunded (crates/contract/src/lib.rs:1496), so 16 NEAR is safe against the ~12.4 NEAR needed at the 1,235,000-byte cap; release tag/asset names and nearone/mpc-node-gcp:<VERSION> match .github/workflows/release.yml:170-214; and file-args with a raw binary has precedent at docs/deploy-tee-verifier.md:111. All prepare-release.sh references are now updated, including the link definition at .github/prompts/pr-review.prompt.md:112.

Changes:

• New scripts/ops/common.sh (errors, dependency checks, semver check, cross-platform sha256, confirm, command echoing) and dev-cluster/dev-common.sh (cluster resolution, endpoint prompts, node build-info verification, test sign).
• migrate-dev-cluster.sh: lists mpc-node* Nomad jobs, rewrites the image via the HTTP API, plans, confirms, registers, waits for the new allocation.
• upgrade-dev-contract.sh: fetches or builds the contract WASM, hand-rolls the borsh encoding, drives propose_update + vote_update per member account.
• menu.sh / dev-menu.sh entry points; prepare-release.sh -> prepare-github-release.sh rename with all references updated.

Reviewed changes

Per-file summary

File	Description
.cliffignore	Updated to the renamed script path.
.github/prompts/pr-review.prompt.md	Rename applied to both the inline reference and the link definition.
RELEASES.md	Rename applied throughout; new "Ops tooling" section describing menu.sh and the dev-cluster flow.
scripts/ops/prepare-release.sh -> prepare-github-release.sh	Renamed; header and usage examples updated.
scripts/ops/common.sh	New generic sourced helpers.
scripts/ops/menu.sh	New top-level interactive menu.
scripts/ops/dev-cluster/dev-common.sh	New dev-cluster helpers.
scripts/ops/dev-cluster/dev-menu.sh	New orchestrator: network/version prompts, then nodes -> verify -> contract.
scripts/ops/dev-cluster/migrate-dev-cluster.sh	New Nomad image-swap step with plan/confirm/apply and allocation wait.
scripts/ops/dev-cluster/upgrade-dev-contract.sh	New contract propose/vote step.

Findings

Blocking (must fix before merge):

• scripts/ops/dev-cluster/upgrade-dev-contract.sh:34 — the release and build sources share one cache path, so a local build can be silently proposed as the released artifact. wasm="${dir}/mpc-contract-v${version}.wasm" is used by both branches: the build branch cps into it (:68), and the release branch short-circuits on [[ -f "$wasm" ]] and prints ==> Reusing ${wasm} (:47-48). The realistic sequence — build locally to smoke-test 3.14.0, then re-run and pick (r)eleased — reuses the local build and proposes it to the cluster, with nothing in the output saying so. The printed sha256 (:83) is the only tell, and there is nothing on screen to compare it against.

  Separate the two, so the reuse shortcut can only ever hit a downloaded artifact:

  local wasm
  if [[ "$source" == release ]]; then
      wasm="${dir}/release/mpc-contract-v${version}.wasm"
  else
      wasm="${dir}/build/mpc-contract-v${version}.wasm"
  fi
  mkdir -p "$(dirname "$wasm")"

Non-blocking (nits, follow-ups, suggestions):

• scripts/ops/dev-cluster/migrate-dev-cluster.sh:75 — wait_for_alloc ends in warn, which returns 0, so upgrade_nomad_job returns cleanly and the loop at :146 moves straight on to the next mpc-node-* job. If node 0's allocation never comes back, the script proceeds to restart node 1, which can drop the cluster below threshold. The per-job confirm at :117 does give the operator a chance to stop, but the failure deserves an explicit gate: confirm " Continue with the remaining jobs?" || die "Stopped after ${job_id}.".
• scripts/ops/dev-cluster/dev-common.sh:121 — the smoke test is new code written against the V1 wire format (payload as a 32-int array + key_version), which crates/contract/README.md:288 documents as the legacy compatibility path and crates/near-mpc-crypto-types/src/sign.rs:47 marks deprecated. It works today, but it will break silently whenever the compat layer is dropped. Prefer the current shape, as in docs/localnet/args/sign_ecdsa.json: {"request": {"path": "test", "domain_id": 0, "payload_v2": {"Ecdsa": "<64 hex chars>"}}}.
• scripts/ops/dev-cluster/upgrade-dev-contract.sh:100 — prepaid-gas '100.0 Tgas' for propose_update is below what the rest of the repo attaches for the same call: crates/near-mpc-contract-interface/src/client.rs:134 uses MAX_GAS (300 Tgas) and the sandbox helper at crates/contract/tests/sandbox/common.rs:362 uses .max_gas(). A ~1.2 MB code payload is dominated by storage-write cost and leaves little headroom at 100 Tgas; unused gas is refunded, so there is no upside to the lower figure. (vote_update at :123 already uses 300.)
• scripts/ops/dev-cluster/migrate-dev-cluster.sh:52 — show_output echoes the whole plan response, and Nomad's plan diff can carry task Env values, which for an MPC node job is exactly where secrets live. Only .FailedTGAllocs and .Warnings are actually consumed (:112-113), so echoing a jq-extracted summary instead of the raw body would avoid putting job secrets in the operator's scrollback. This also sits oddly next to the comment at :22 ("Credentials are never part of what's printed"), which is about the curl credentials only.
• scripts/ops/dev-cluster/migrate-dev-cluster.sh:31 — the echoed command is not the one that runs: it prints --data @- while the real call passes --data "$data" (:29) and pipes the curl config on stdin, and it omits -H Content-Type: application/json. Copy-pasting the printed line would hang on stdin. The same mismatch is at scripts/ops/dev-cluster/dev-common.sh:99 versus :101-102 (--max-time 5 and grep -o ... are dropped). RELEASES.md:180 sells "Every command is printed before it runs", so these are worth keeping honest.
• scripts/ops/dev-cluster/dev-menu.sh:79 — upgrade-dev-contract.sh exits 0 when the operator declines at :106 ("Aborted before proposing.") or skips every vote, so the summary reports contract: upgraded for a run that changed nothing. Having the child exit non-zero on an abort, or printing a distinct "declined" state, would make the final line trustworthy.
• RELEASES.md:181 — "Nothing cluster-specific is stored in this repo" is contradicted by scripts/ops/dev-cluster/dev-common.sh:20-25, which hardcodes mpc-dev-contract.testnet, dev-contract.near, and all four member account IDs. Narrowing it to "no endpoints or credentials are stored in this repo" would match what the code does.
• scripts/ops/dev-cluster/dev-common.sh:53 — a bare IP becomes http://${input}, i.e. port 80, while Nomad's HTTP API defaults to 4646. The prompt text does show the :4646 form, but defaulting the port when none is given ([[ "$input" == *:* ]] || input="${input}:4646") would remove a likely first-run stumble.
• scripts/ops/dev-cluster/migrate-dev-cluster.sh:36 — config+="user = \"${NOMAD_HTTP_AUTH}\"" breaks if the password contains a " or \, since curl config values are quoted strings with backslash escapes. A password with a quote would produce wrong credentials rather than an error. Escaping both characters before interpolating would make it robust.

⚠️ Issues found

[SimonRastikian]

I decided to split this PR into 3 smaller ones