scripts: add run_in_parallel helper for prod ops

[matanl-starkware]

Add a thread-pool-based run_in_parallel helper to common_lib that buffers each
worker's output into a labeled block, prints a 5s progress heartbeat naming
still-running items, and aggregates per-item failures into a single non-zero
exit. Resolve print_colored's output stream at call time so redirection works.

Foundation for parallelizing per-node kubectl operations; nothing calls it yet.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

[cursor]

PR Summary

Low Risk
New library code and tests only; no callers yet, so production kubectl flows are unchanged until follow-up wiring.

Overview
Adds run_in_parallel to common_lib for capped thread-pool execution of I/O-bound prod work (e.g. future per-node kubectl), with per-worker log buffering via thread-local _output_sink, labeled output blocks on completion, periodic heartbeats for still-running items, and aggregated failures with sys.exit(1) after all items finish.

print_colored now resolves file at call time (fixes redirection) and appends to the worker buffer when parallel capture is active; a _print_lock keeps real stdout/stderr writes from interleaving.

New test_run_in_parallel.py covers result ordering, empty input, buffered output grouping, heartbeats, and failure reporting. No prod scripts call the helper yet—library-only foundation.

^{Reviewed by Cursor Bugbot for commit df94c18. Bugbot is set up for automated code reviews on this repo. Configure here.}

[matanl-starkware]

• scripts: make --service a required argument #14306
• scripts: confirm before restarting all Core pods at once #14305
• scripts: add Committer to Service options #14304
• scripts: clean up restart-strategy and service help output #14303
• scripts: restart and wait on nodes in parallel #14302
• scripts: make metrics gate() thread-safe #14301
• scripts: add run_in_parallel helper for prod ops #14300 👈 (View in Graphite)
• ci: fix hybrid system test for prod-script changes and k3d install #14317
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[reviewable-StarkWare]

This change is 

[ron-starkware]

@ron-starkware reviewed 2 files and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on matanl-starkware).

---

scripts/prod/common_lib.py line 77 at r1 (raw file):

    is printed every `heartbeat_interval_seconds`.

    Errors: workers must raise rather than call `sys.exit()` (a `SystemExit` raised in a worker

I see that except BaseException does catch sys.exit(), which the workers throw on an exception (and then prints the SystemExit code).

---

scripts/prod/common_lib.py line 91 at r1 (raw file):

    errors: dict[int, BaseException] = {}

    def run_one(index: int, item: T) -> R:

I see that index isn't used in this function (nor later in the stack)

[matanl-starkware]

Addressed in the latest push:

• Removed the unused index parameter from run_one.
• run_in_parallel now re-raises KeyboardInterrupt instead of recording it as an item failure, so Ctrl-C aborts the whole run (per Cursor Bugbot).
• Reworded the docstring to match behavior: a worker that raises or calls sys.exit() is recorded and aggregated into a single non-zero exit; only KeyboardInterrupt propagates.

[ron-starkware]

@ron-starkware reviewed 1 file and all commit messages, and resolved 2 discussions.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on matanl-starkware).

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 04009a0. Configure here.}

[matanl-starkware]

@matanl-starkware resolved 2 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on matanl-starkware).

[graphite-app]

Merge activity

• Jun 3, 12:20 PM UTC: Graphite rebased this pull request, because this pull request is set to merge when ready.