scripts: make metrics gate() thread-safe

[matanl-starkware]

Guard signal.signal registration to the main thread (it raises ValueError on
worker threads), and track live kubectl port-forward processes in a lock-guarded
registry exposing terminate_all_port_forwards for best-effort cleanup from a
main-thread signal handler. Enables gate() to run under parallel waits.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

[cursor]

PR Summary

Low Risk
Prod scripting/metrics wait tooling only; no auth, data, or runtime service paths.

Overview
Makes MetricConditionGater.gate() safe to run from worker threads during parallel metric waits.

Adds a lock-protected registry of live kubectl port-forward processes and terminate_all_port_forwards() so a main-thread SIGINT/SIGTERM handler can tear down all forwards when per-thread signal.signal is unavailable. gate() registers each forward on start and removes it in _terminate_port_forward_process. Signal handlers are installed only on the main thread; worker threads rely on the orchestrator’s handler plus the existing finally cleanup.

Replaces the startup assert with a check that treats an early exit as a clean return when _shutdown_requested is set (interrupt during startup), otherwise raises RuntimeError. Status messages use print_colored.

^{Reviewed by Cursor Bugbot for commit a69f19c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[reviewable-StarkWare]

This change is 

[matanl-starkware]

• scripts: make --service a required argument #14306
• scripts: confirm before restarting all Core pods at once #14305
• scripts: add Committer to Service options #14304
• scripts: clean up restart-strategy and service help output #14303
• scripts: restart and wait on nodes in parallel #14302
• scripts: make metrics gate() thread-safe #14301 👈 (View in Graphite)
• scripts: add run_in_parallel helper for prod ops #14300
• ci: fix hybrid system test for prod-script changes and k3d install #14317
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[ron-starkware]

@ron-starkware reviewed 1 file and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on matanl-starkware).

---

scripts/prod/metrics_lib.py line 168 at r1 (raw file):

            with _active_port_forwards_lock:
                _active_port_forwards.add(pf_process)
            print("Waiting for forwarding to start")

Should this be changed to print_colored instead so that it doesn't write to the shared stdout?

Suggestion:

print_colored("Waiting for forwarding to start")

---

scripts/prod/metrics_lib.py line 178 at r1 (raw file):

            print(
                f"Forwarding started (from local port {self.local_port} to {self.pod}:{self.metrics_port})"
            )

Should this be changed to print_colored instead so that it doesn't write to the shared stdout?

Suggestion:

            print_colored(
                f"Forwarding started (from local port {self.local_port} to {self.pod}:{self.metrics_port})"
            )

[matanl-starkware]

Addressed: both bare print(...) calls in gate() ("Waiting for forwarding to start" and "Forwarding started ...") now use print_colored(...) so they honor the per-node output buffer under parallel waits.

[ron-starkware]

@ron-starkware reviewed 1 file and all commit messages, and resolved 2 discussions.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on matanl-starkware).

[matanl-starkware]

@ron-starkware — could you re-approve this one when you get a chance? 🙏

Context: #14300 (bottom of the stack) merged, so I rebased the rest onto main. The rebase reset Reviewable's status check, so it needs a fresh LGTM on this revision.

I also addressed the new Cursor Bugbot finding on this push: in gate() on a worker thread, a Ctrl-C handler calling terminate_all_port_forwards during the 3s startup sleep could trip the assert pf_process.poll() is None (teardown-only). It now records a shutdown flag and gate() returns cleanly in that case instead of asserting; a genuine startup failure raises a descriptive RuntimeError instead of a bare assert.

Heads-up on cadence: because the repo squash-merges and these are separate stacked PRs, each time I merge one and gt sync, the next PR rebases and its Reviewable check resets — so I'll request re-approval one at a time (#14302 after this merges, etc.). Appreciate the patience!

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a69f19c. Configure here.}

[ron-starkware]

@ron-starkware reviewed 1 file and all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on matanl-starkware).

[matanl-starkware]

@matanl-starkware resolved 2 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on matanl-starkware).