github/gen-matrix.py: add sanitize mode, exclude g++ debug+sanitize

Add 'sanitize' as a third value in MODES so all-pairs covers it across
the compiler/standard/arch axes instead of pinning it to two explicit
one-offs. The regular grid grows from 8 to 10 items; specials drop by
two for a net +2 rows.

Exclude (g++-{15,16}, debug) and (g++-{15,16}, sanitize) from the
regular matrix via allpairspy's filter_func. gcc 15+ miscompiles
structured bindings inside loops in coroutines: the hidden tuple's
lifetime tracking is broken, so the destructor runs on uninitialized
(ASan-poisoned, 0xBE) stack memory. In debug mode this surfaces as a
pollable_fd_state LSan leak via reactor::do_accept's cross-shard
forwarding path; in sanitize mode UBSan aborts the rpc client/server
loops at startup. Drop EXCLUDED_PAIRS once
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=124584 is fixed (tracked
in scylladb#3431) so g++ debug + sanitize come back into
coverage.

Also emit tests.yaml via ruamel.yaml instead of hand-rolled string
formatting (PR review feedback): load the workflow, replace the
strategy matrix include list in place by key path, and dump it back so
the library handles scalar quoting while the rest of the file
round-trips untouched. Drops the _yaml_val/_format_item/_NEEDS_QUOTE
helpers and the BEGIN/END text-splice markers, and adds ruamel.yaml to
the inline-script dependencies. Add the Apache license header to
gen-matrix.py.

Author: travisdowns

.github/workflows/gen-matrix.py

@@ -1,32 +1,53 @@
 #!/usr/bin/env -S uv run --script
+#
+# This file is open source software, licensed to you under the terms
+# of the Apache License, Version 2.0 (the "License").  See the NOTICE file
+# distributed with this work for additional information regarding copyright
+# ownership.  You may not use this file except in compliance with the License.
+#
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Copyright (C) 2026 Redpanda Data.
+#
 # /// script
 # requires-python = ">=3.10"
 # dependencies = [
 #   "allpairspy",
+#   "ruamel.yaml",
 # ]
 # ///
 """
 Generate the regular_test include: matrix for tests.yaml.
 
-Uses all-pairs (pairwise) coverage over compiler x standard x mode for
-the regular builds, plus an explicit list of special-purpose jobs
-(dpdk, cxx-modules, fuzz). All-pairs guarantees that every pair of
-parameter values is exercised by at least one job, with far fewer
-combinations than the full cartesian product (12 vs 24 for our 4x2x3
-matrix). Everything outside the marked region of tests.yaml is left
-untouched.
+Uses all-pairs (pairwise) coverage over compiler x standard x mode x
+arch for the regular builds, plus an explicit list of special-purpose
+jobs (dpdk, cxx-modules, fuzz). All-pairs guarantees that every pair
+of parameter values is exercised by at least one job, with far fewer
+combinations than the full cartesian product. Excluded pairs (see
+EXCLUDED_PAIRS) drop out of regular coverage. The file is round-tripped
+through ruamel.yaml, replacing only the strategy matrix include list;
+everything else in tests.yaml is left untouched.
 
 Usage:
     .github/workflows/gen-matrix.py
     git diff .github/workflows/tests.yaml
 """
 
-from __future__ import annotations
-
 import pathlib
 from typing import Any
 
 from allpairspy import AllPairs
+from ruamel.yaml import YAML
+from ruamel.yaml.comments import CommentedMap, CommentedSeq
 
 HERE = pathlib.Path(__file__).parent
 TESTS_YAML = HERE / "tests.yaml"
@@ -42,12 +63,33 @@
     + [f"g++-{v}"   for v in GCC_VERSIONS]
 )
 STANDARDS: list[int] = [20, 23]
-MODES: list[str] = ["debug", "release"]
+MODES: list[str] = ["debug", "release", "sanitize"]
 ARCHS: list[str] = ["x86", "arm"]
 
-# Markers that bracket the managed region inside tests.yaml.
-BEGIN_TAG = "# This include block is AUTOGENERATED"
-END_TAG = "# end of AUTOGENERATED part"
+# Compiler+mode combinations to skip in the regular all-pairs matrix.
+# gcc 15+ miscompiles structured bindings inside loops in coroutines
+# (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=124584), breaking g++
+# debug (LSan: pollable_fd_state leak via cross-shard accept) and g++
+# sanitize (UBSan abort on poisoned rcv_buf in rpc loops). Tracked in
+# scylladb/seastar#3431.
+EXCLUDED_PAIRS: list[tuple[str, str]] = [
+    (f"g++-{v}", m) for v in GCC_VERSIONS for m in ("debug", "sanitize")
+]
+
+
+def _keep_row(row: list[Any]) -> bool:
+    """allpairspy filter_func: drop rows hitting an EXCLUDED_PAIRS entry.
+
+    Called with progressively longer prefixes during pair generation, so
+    the check only fires once both compiler (row[0]) and mode (row[2])
+    are present in the partial row.
+    """
+    return not (len(row) >= 3 and (row[0], row[2]) in EXCLUDED_PAIRS)
+
+# Key path to the strategy matrix map inside tests.yaml; we replace its
+# "include" list. The "# AUTOGENERATED" comment is preserved automatically
+# on round-trip.
+MATRIX_PATH = ("jobs", "regular_test", "strategy", "matrix")
 
 # Special jobs merged into the same matrix so tests.yaml stays a single job.
 # Items only set enable-ccache when overriding the default (test.yaml treats
@@ -86,53 +128,50 @@
     },
 ]
 
-# Characters that require a value to be double-quoted in YAML flow style.
-_NEEDS_QUOTE: frozenset[str] = frozenset(' ,:{}"[]#')
+def _build_include(items: list[dict[str, Any]]) -> CommentedSeq:
+    """Build the matrix include list as a block sequence of flow mappings.
 
-
-def _yaml_val(v: object) -> str:
-    if isinstance(v, bool):
-        return "true" if v else "false"
-    if isinstance(v, int):
-        return str(v)
-    s = str(v)
-    if any(c in s for c in _NEEDS_QUOTE):
-        return '"' + s.replace("\\", "\\\\").replace('"', '\\"') + '"'
-    return s
-
-
-def _format_item(item: dict[str, Any], indent: str) -> str:
-    parts = ", ".join(f"{k}: {_yaml_val(v)}" for k, v in item.items())
-    return f"{indent}  - {{{parts}}}\n"
+    Each item becomes a flow-style mapping (one job per line) so the block
+    stays compact and diff-friendly; ruamel.yaml handles scalar quoting.
+    """
+    seq = CommentedSeq()
+    for item in items:
+        mapping = CommentedMap(item)
+        mapping.fa.set_flow_style()
+        seq.append(mapping)
+    return seq
 
 
 def generate() -> list[dict[str, Any]]:
     regular: list[dict[str, Any]] = [
         {"compiler": c, "standard": s, "mode": m, "arch": a}
-        for c, s, m, a in AllPairs([COMPILERS, STANDARDS, MODES, ARCHS])
+        for c, s, m, a in AllPairs(
+            [COMPILERS, STANDARDS, MODES, ARCHS],
+            filter_func=_keep_row,
+        )
     ]
     return regular + SPECIAL_ITEMS
 
 
 def main() -> None:
     items = generate()
-    lines: list[str] = TESTS_YAML.read_text().splitlines(keepends=True)
 
-    begin_idx = next(i for i, line in enumerate(lines) if BEGIN_TAG in line)
-    end_idx   = next(i for i, line in enumerate(lines) if END_TAG   in line)
+    yaml = YAML()
+    yaml.preserve_quotes = True
+    yaml.width = 1 << 20  # never wrap a flow mapping across lines
+    yaml.indent(mapping=2, sequence=4, offset=2)
 
-    first = lines[begin_idx]
-    indent = " " * (len(first) - len(first.lstrip()))
+    data = yaml.load(TESTS_YAML)
+    matrix = data
+    for key in MATRIX_PATH:
+        matrix = matrix[key]
 
-    new_lines: list[str] = [
-        f"{indent}{BEGIN_TAG}, do not edit by hand\n",
-        f"{indent}# instead edit + re-run .github/workflows/gen-matrix.py\n",
-        f"{indent}include:\n",
-    ]
-    for item in items:
-        new_lines.append(_format_item(item, indent))
+    # We only swap out the include list. The "# AUTOGENERATED, do not edit"
+    # comment is attached to the include key, not the list, so it is
+    # preserved automatically on round-trip.
+    matrix["include"] = _build_include(items)
 
-    TESTS_YAML.write_text("".join(lines[:begin_idx] + new_lines + lines[end_idx:]))
+    yaml.dump(data, TESTS_YAML)
     print(f"Updated {TESTS_YAML}: {len(items)} matrix items")
 
 

.github/workflows/tests.yaml

@@ -24,17 +24,17 @@ jobs:
         include:
           - {compiler: clang++-21, standard: 20, mode: debug, arch: x86}
           - {compiler: clang++-22, standard: 23, mode: release, arch: x86}
-          - {compiler: g++-15, standard: 23, mode: debug, arch: arm}
+          - {compiler: g++-15, standard: 23, mode: release, arch: arm}
           - {compiler: g++-16, standard: 20, mode: release, arch: arm}
-          - {compiler: g++-16, standard: 23, mode: debug, arch: x86}
+          - {compiler: g++-16, standard: 23, mode: release, arch: x86}
           - {compiler: g++-15, standard: 20, mode: release, arch: x86}
-          - {compiler: clang++-22, standard: 20, mode: debug, arch: arm}
-          - {compiler: clang++-21, standard: 23, mode: release, arch: arm}
+          - {compiler: clang++-22, standard: 20, mode: sanitize, arch: arm}
+          - {compiler: clang++-21, standard: 23, mode: sanitize, arch: x86}
+          - {compiler: clang++-21, standard: 23, mode: debug, arch: arm}
           - {compiler: clang++-22, standard: 23, arch: x86, mode: dev}
-          - {compiler: clang++-22, standard: 23, arch: x86, mode: release, enables: --enable-dpdk, options: "--cook dpdk --dpdk-machine corei7-avx", info: "dpdk, "}
-          - {compiler: clang++-22, standard: 23, arch: x86, mode: debug, enables: --enable-cxx-modules, enable-ccache: false, info: "modules, "}
-          - {compiler: clang++-22, standard: 23, arch: x86, mode: fuzz, test-args: "-- -R 'Seastar.fuzz.'"}
-        # end of AUTOGENERATED part
+          - {compiler: clang++-22, standard: 23, arch: x86, mode: release, enables: --enable-dpdk, options: --cook dpdk --dpdk-machine corei7-avx, info: 'dpdk, '}
+          - {compiler: clang++-22, standard: 23, arch: x86, mode: debug, enables: --enable-cxx-modules, enable-ccache: false, info: 'modules, '}
+          - {compiler: clang++-22, standard: 23, arch: x86, mode: fuzz, test-args: -- -R 'Seastar.fuzz.'}
     with:
       compiler: ${{ matrix.compiler }}
       standard: ${{ matrix.standard }}