[ez][alerts] Increase queue alert threshold to 150 machines (#7024)

Queue alerts fire too frequently, a queue of 75 linux 2x large is not
crazy, as long as its a short time. The thresholds need refinement, but
I think this is a safe way to reduce the number of alerts. A queue of
150 linux 2x large is probably an issue? maybe

Author: clee2000

tools/torchci/queue_alert.py

@@ -20,7 +20,7 @@
 QUEUE_ALERT_LABEL = "queue-alert"
 
 MAX_HOURS = 4
-MAX_MACHINES = 75
+MAX_MACHINES = 150
 EXCEPTIONS = {"linux.gcp.a100.large": (20, 20)}
 
 

tools/torchci/tests/test_queue_alert.py

@@ -8,7 +8,7 @@ class TestGitHubPR(TestCase):
     def test_filter_long_queues(self):
         db_results = [
             {"count": 30, "avg_queue_s": 0, "machine_type": "linux.gcp.a100.large"},
-            {"count": 100, "avg_queue_s": 0, "machine_type": "machine1"},
+            {"count": 200, "avg_queue_s": 0, "machine_type": "machine1"},
             {"count": 30, "avg_queue_s": 3600 * 5, "machine_type": "machine2"},
         ]
         long_queues = filter_long_queues(db_results)