[vllm-triage] Detect test-level regressions in jobs that fail on both twins (#8521)

This PR fills a gap in the new error detection

The following will now be covered with the following behavior:
If Job A fails in vllm pytorch nightly with tests 1/2 failing and Job A
fails in vllm full ci with test 1, then new error (test 2) is never
detected and nothing is pulled.

Now Job A test 2 will come up with as a new failure. We know this
without looking at the exception chain.

In addition if a new failure from the above is detected. We record the
exception chains of the shared test failures for that job test 1 and
will let the agent determine if there are any additional failures for
that job.

Author: morrison-turnansky

.github/workflows/vllm-torch-nightly-triage.yml

@@ -170,6 +170,13 @@ jobs:
               - cluster-logs/*.log  one representative Buildkite log per cluster,
                 ANSI-stripped and tail-trimmed. The first lines of each file give the
                 cluster name, job name, job URL, state and exit status.
+                Files prefixed `both_` are red-on-both-twins clusters (see report.json
+                "regressed_tests"): the job fails on both the torch-nightly and
+                baseline build, but fails MORE tests on nightly. A "shared failure(s)"
+                section pairs each shared failure's torch_nightly_exception_chain and
+                baseline_exception_chain so you can judge whether a shared failure
+                changed. Only the torch-nightly-only tests listed above that section
+                are definitely new.
 
             For each cluster, identify the actual failure: the failed test IDs and the
             real exception. Remember that "Engine core initialization failed. See root

tools/torchci/tests/test_vllm_log_parser.py

@@ -7,7 +7,7 @@
 import unittest
 from pathlib import Path
 
-from torchci.vllm_log_parser import parse_log, strip_markers
+from torchci.vllm_log_parser import get_test_signature, parse_log, strip_markers
 
 
 FIXTURES_DIR = Path(__file__).parent / "fixtures"
@@ -889,5 +889,24 @@ def test_undefined_symbol_not_tagged(self) -> None:
         self.assertFalse(parsed_log.job_is_infra)
 
 
+class TestGetTestSignature(unittest.TestCase):
+    """Signature is the (test_id, pytest_exception_class) 2-tuple."""
+
+    def test_signature_is_id_and_class(self) -> None:
+        log = (
+            "FAILED tests/test_a.py::test_one - ValueError: bad\n= 1 failed in 1.00s ="
+        )
+        failure = parse_log(log).pytest_results[0].test_failures[0]
+        self.assertEqual(
+            get_test_signature(failure),
+            ("tests/test_a.py::test_one", "ValueError"),
+        )
+
+    def test_signature_empty_class_for_bare_failed(self) -> None:
+        log = "FAILED tests/test_a.py::test_one\n= 1 failed in 1.00s ="
+        failure = parse_log(log).pytest_results[0].test_failures[0]
+        self.assertEqual(get_test_signature(failure), ("tests/test_a.py::test_one", ""))
+
+
 if __name__ == "__main__":
     unittest.main()