scripts: restart and wait on nodes in parallel

[matanl-starkware]

For the non-interactive ALL_AT_ONCE strategy, restart every node's pod and then
run the post-restart health/metric waits concurrently instead of node-by-node,
which was the main source of slow rollouts. Add ServiceRestarter.restart_all
driving two parallel phases (restart, then wait) via run_in_parallel, gated by a
new --max-parallelism flag (default 16). ONE_BY_ONE and NO_RESTART stay
sequential since they prompt the user between nodes. Thread the flag through all
four entry scripts. Also fix _wait_for_pod_to_satisfy_condition to return True on
success (it returned None, so callers always logged a spurious failure).

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

[cursor]

PR Summary

Medium Risk
Touches production rollout scripts that delete pods and gate on metrics across many nodes at once; mis-tuned parallelism or signal handling could complicate incident response, though interactive strategies are unchanged.

Overview
For ALL_AT_ONCE, prod restart scripts now restart every node’s pod in parallel, then run post-restart waits (pod ready + metric gates) in parallel via new ServiceRestarter.restart_all and run_in_parallel, instead of finishing one node before starting the next.

A --max-parallelism / -p flag (default 16) caps concurrent restarts/waits; ONE_BY_ONE and NO_RESTART stay sequential. The flag is threaded through update_config_and_restart_nodes and the revert, observer, config-update, and proposal-wait entry scripts. Kubectl output is captured so logs stay readable under parallelism; Ctrl-C on metric waits calls terminate_all_port_forwards. _wait_for_pod_to_satisfy_condition now returns True on success (fixing spurious failure logs). Unit tests in test_parallel_restart.py assert concurrency behavior.

^{Reviewed by Cursor Bugbot for commit 852f831. Bugbot is set up for automated code reviews on this repo. Configure here.}

[reviewable-StarkWare]

This change is 

[matanl-starkware]

• scripts: make --service a required argument #14306
• scripts: confirm before restarting all Core pods at once #14305
• scripts: add Committer to Service options #14304
• scripts: clean up restart-strategy and service help output #14303
• scripts: restart and wait on nodes in parallel #14302 👈 (View in Graphite)
• scripts: make metrics gate() thread-safe #14301
• scripts: add run_in_parallel helper for prod ops #14300
• ci: fix hybrid system test for prod-script changes and k3d install #14317
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 244baa3. Configure here.}

[ron-starkware]

@ron-starkware reviewed 7 files and all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on matanl-starkware).

[matanl-starkware]

Re: the parallel restart sys.exit note — this is intended for the ALL_AT_ONCE flow: pod deletes are dispatched concurrently, and a failure in any worker is collected by run_in_parallel and surfaced as a non-zero exit once the in-flight operations settle (it is not silently swallowed). Aborting mid-flight isn't meaningful once deletes are already dispatched in parallel; the sequential one_by_one strategy retains the immediate-abort-on-failure behavior. (KeyboardInterrupt now propagates, per the helper PR.)

[matanl-starkware]

@ron-starkware — next in the stack 🙏 #14301 merged, so I rebased this onto main and its Reviewable check reset. Could you re-approve this revision when you have a moment? (No code changes from your prior approval — just the rebase.)