test: make TEST_TAP_TIMEOUT actually fire, and enable it by default

TEST_TAP_TIMEOUT could not catch the failure it exists for. The read loop
was:

    line = fop.stdout.readline()          # blocks
    ...
    if tap_timeout > 0 and (time.time() - start_time) > tap_timeout:

readline() blocks until a full line arrives, so a test that hangs while
producing no output never reached the deadline check at all. Verified
directly: with tap_timeout=3 against 'sleep 300', the old loop was still
blocked after 25 seconds; the new one raises at 3.0s.

That is the exact profile of the CI-mysql84-g9 stall on #5991, where
test_ssl_fast_forward-3_libmariadb-t ran for hours.

Replaces the blocking readline() with select() on a bounded wait plus raw
os.read() chunking. select() guarantees the deadline check runs even when
the child is silent; chunking rather than line-reading means a test that
stops mid-line cannot wedge the loop either. Output order and content are
unchanged, a trailing partial line is now flushed instead of dropped, and
decoding uses errors='replace' so a stray non-UTF-8 byte no longer throws.

Verified against four cases: normal chatty test (all lines, in order),
silent hang (timeout fires), partial-line-then-hang (timeout fires), and
clean exit.

Also flips the default from 0 (disabled) to 1800s. 1800 is ~2.4x the
slowest single test measured across 47 groups:

    reg_test_3765_ssl_pollout-t   12.5 min
    test_cluster_sync-t           10.5 min
    set_testing-240-t              7.8 min
    test_auth_methods-t            7.7 min

Only 4 of 401 tests exceed 5 minutes, so this cannot fire on a merely slow
test, while still stopping a hang well inside the 90-minute step budget
added in the companion CI PRs -- and, unlike a step or job timeout, it
identifies WHICH test hung and lets the run continue to its archive steps.

Author: renecannao

test/infra/control/env-isolated.bash

@@ -104,7 +104,14 @@ export TEST_PY_TAP_REPEAT="${TEST_PY_TAP_REPEAT:-1}"
 export TEST_PY_TAP_SHUFFLE_LIMIT="${TEST_PY_TAP_SHUFFLE_LIMIT:-0}"
 export TEST_PY_TAP_DUMP_RUNTIME="${TEST_PY_TAP_DUMP_RUNTIME:-1}"
 export TEST_PY_TAP_DUMP_STATS="${TEST_PY_TAP_DUMP_STATS:-1}"
-export TEST_TAP_TIMEOUT="${TEST_TAP_TIMEOUT:-0}"
+# Per-test wall-clock budget, in seconds. 0 disables it entirely, which was
+# the previous default: a hung TAP test then ran until the CI job itself was
+# killed. 1800 is ~2.4x the slowest single test measured across 47 groups
+# (reg_test_3765_ssl_pollout-t, 12.5 min; then test_cluster_sync-t 10.5,
+# set_testing-240-t 7.8, test_auth_methods-t 7.7 -- only 4 tests exceed 5
+# minutes at all), so it cannot fire on a merely slow test while still
+# catching a hang long before the 90-minute step budget.
+export TEST_TAP_TIMEOUT="${TEST_TAP_TIMEOUT:-1800}"
 
 # Cluster sync test support — expose first cluster node admin port for replica validation
 if [ "${NUM_CLUSTER_NODES}" -gt 0 ]; then

test/scripts/bin/proxysql-tester.py

@@ -5,6 +5,7 @@
 import os
 import pymysql
 import sys
+import select
 import subprocess
 import random
 import time
@@ -842,19 +843,42 @@ def disk_usage():
                         sys.exit(1)
                     continue
 
-                # Run test with timeout if specified
+                # Run test with timeout if specified.
+                #
+                # The read has to be non-blocking for tap_timeout to mean
+                # anything. readline() blocks until a full line arrives, so on
+                # a test that hangs while producing no output -- precisely the
+                # case this timeout exists to catch -- the deadline check below
+                # it was simply never reached, and the test ran until CI killed
+                # the job. select() bounds the wait so the check always runs,
+                # and reading raw chunks rather than lines means a test that
+                # stops mid-line cannot wedge us either.
                 try:
                     start_time = time.time()
+                    buf = b''
                     while True:
-                        line = fop.stdout.readline()
-                        if not line and fop.poll() is not None:
-                            break
-                        if line:
-                            log.debug(f"msg: {line.decode('utf-8').strip()}")
-                        
                         if tap_timeout > 0 and (time.time() - start_time) > tap_timeout:
                             raise subprocess.TimeoutExpired(fop.args, tap_timeout)
-                    
+
+                        wait = 1.0
+                        if tap_timeout > 0:
+                            wait = max(0.0, min(1.0, tap_timeout - (time.time() - start_time)))
+                        ready, _, _ = select.select([fop.stdout], [], [], wait)
+
+                        if ready:
+                            chunk = os.read(fop.stdout.fileno(), 65536)
+                            if not chunk:
+                                break                     # EOF: test finished
+                            buf += chunk
+                            while b'\n' in buf:
+                                line, buf = buf.split(b'\n', 1)
+                                log.debug(f"msg: {line.decode('utf-8', 'replace').strip()}")
+                        elif fop.poll() is not None:
+                            break
+
+                    if buf:                               # trailing partial line
+                        log.debug(f"msg: {buf.decode('utf-8', 'replace').strip()}")
+
                     fop.wait()
                 except subprocess.TimeoutExpired:
                     fop.kill()