github/gen-matrix.py: add sanitize mode, exclude g++ debug+sanitize

Add 'sanitize' as a third value in MODES so all-pairs covers it across
the compiler/standard/arch axes instead of pinning it to two explicit
one-offs. The regular grid grows from 8 to 10 items; specials drop by
two for a net +2 rows.

Exclude (g++-{15,16}, debug) and (g++-{15,16}, sanitize) from the
regular matrix via allpairspy's filter_func. gcc 15+ miscompiles
structured bindings inside loops in coroutines: the hidden tuple's
lifetime tracking is broken, so the destructor runs on uninitialized
(ASan-poisoned, 0xBE) stack memory. In debug mode this surfaces as a
pollable_fd_state LSan leak via reactor::do_accept's cross-shard
forwarding path; in sanitize mode UBSan aborts the rpc client/server
loops at startup. Drop EXCLUDED_PAIRS once
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=124584 is fixed (tracked
in scylladb#3431) so g++ debug + sanitize come back into
coverage.

Author: travisdowns

.github/workflows/gen-matrix.py

@@ -8,13 +8,13 @@
 """
 Generate the regular_test include: matrix for tests.yaml.
 
-Uses all-pairs (pairwise) coverage over compiler x standard x mode for
-the regular builds, plus an explicit list of special-purpose jobs
-(dpdk, cxx-modules, fuzz). All-pairs guarantees that every pair of
-parameter values is exercised by at least one job, with far fewer
-combinations than the full cartesian product (12 vs 24 for our 4x2x3
-matrix). Everything outside the marked region of tests.yaml is left
-untouched.
+Uses all-pairs (pairwise) coverage over compiler x standard x mode x
+arch for the regular builds, plus an explicit list of special-purpose
+jobs (dpdk, cxx-modules, fuzz). All-pairs guarantees that every pair
+of parameter values is exercised by at least one job, with far fewer
+combinations than the full cartesian product. Excluded pairs (see
+EXCLUDED_PAIRS) drop out of regular coverage. Everything outside the
+marked region of tests.yaml is left untouched.
 
 Usage:
     .github/workflows/gen-matrix.py
@@ -42,9 +42,29 @@
     + [f"g++-{v}"   for v in GCC_VERSIONS]
 )
 STANDARDS: list[int] = [20, 23]
-MODES: list[str] = ["debug", "release"]
+MODES: list[str] = ["debug", "release", "sanitize"]
 ARCHS: list[str] = ["x86", "arm"]
 
+# Compiler+mode combinations to skip in the regular all-pairs matrix.
+# gcc 15+ miscompiles structured bindings inside loops in coroutines
+# (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=124584), breaking g++
+# debug (LSan: pollable_fd_state leak via cross-shard accept) and g++
+# sanitize (UBSan abort on poisoned rcv_buf in rpc loops). Tracked in
+# scylladb/seastar#3431.
+EXCLUDED_PAIRS: list[tuple[str, str]] = [
+    (f"g++-{v}", m) for v in GCC_VERSIONS for m in ("debug", "sanitize")
+]
+
+
+def _keep_row(row: list[Any]) -> bool:
+    """allpairspy filter_func: drop rows hitting an EXCLUDED_PAIRS entry.
+
+    Called with progressively longer prefixes during pair generation, so
+    the check only fires once both compiler (row[0]) and mode (row[2])
+    are present in the partial row.
+    """
+    return not (len(row) >= 3 and (row[0], row[2]) in EXCLUDED_PAIRS)
+
 # Markers that bracket the managed region inside tests.yaml.
 BEGIN_TAG = "# This include block is AUTOGENERATED"
 END_TAG = "# end of AUTOGENERATED part"
@@ -109,7 +129,10 @@ def _format_item(item: dict[str, Any], indent: str) -> str:
 def generate() -> list[dict[str, Any]]:
     regular: list[dict[str, Any]] = [
         {"compiler": c, "standard": s, "mode": m, "arch": a}
-        for c, s, m, a in AllPairs([COMPILERS, STANDARDS, MODES, ARCHS])
+        for c, s, m, a in AllPairs(
+            [COMPILERS, STANDARDS, MODES, ARCHS],
+            filter_func=_keep_row,
+        )
     ]
     return regular + SPECIAL_ITEMS
 

.github/workflows/tests.yaml

@@ -24,12 +24,13 @@ jobs:
         include:
           - {compiler: clang++-21, standard: 20, mode: debug, arch: x86}
           - {compiler: clang++-22, standard: 23, mode: release, arch: x86}
-          - {compiler: g++-15, standard: 23, mode: debug, arch: arm}
+          - {compiler: g++-15, standard: 23, mode: release, arch: arm}
           - {compiler: g++-16, standard: 20, mode: release, arch: arm}
-          - {compiler: g++-16, standard: 23, mode: debug, arch: x86}
+          - {compiler: g++-16, standard: 23, mode: release, arch: x86}
           - {compiler: g++-15, standard: 20, mode: release, arch: x86}
-          - {compiler: clang++-22, standard: 20, mode: debug, arch: arm}
-          - {compiler: clang++-21, standard: 23, mode: release, arch: arm}
+          - {compiler: clang++-22, standard: 20, mode: sanitize, arch: arm}
+          - {compiler: clang++-21, standard: 23, mode: sanitize, arch: x86}
+          - {compiler: clang++-21, standard: 23, mode: debug, arch: arm}
           - {compiler: clang++-22, standard: 23, arch: x86, mode: dev}
           - {compiler: clang++-22, standard: 23, arch: x86, mode: release, enables: --enable-dpdk, options: "--cook dpdk --dpdk-machine corei7-avx", info: "dpdk, "}
           - {compiler: clang++-22, standard: 23, arch: x86, mode: debug, enables: --enable-cxx-modules, enable-ccache: false, info: "modules, "}