Skip to content

Commit d40304d

Browse files
RelativisticJetRelativisticJet
committed
fix(mutmut): wire MUTATE_PATH->TEST_RUNNER_FILES mapping + hard-fail on unknown
v1.1 item A from the 2026-05-18 pre-release sweep — closes the misconfiguration that produced 45 phantom wl_csv.py survivors. ## Problem Prior scripts/mutmut.sh defaulted TEST_RUNNER_FILES to the wl_validation test set regardless of MUTATE_PATH. When a contributor ran with MUTATE_PATH=bin/wl_csv.py (or any non-default module) without also overriding TEST_RUNNER_FILES, mutmut applied mutations to wl_csv.py while running tests that never imported wl_csv. Every mutation trivially survived — not because of a real coverage gap but because the tests didn't exercise the mutated code at all. The 2026-05-18 session caught 45 wl_csv 'survivors' that were entirely artifacts of this misconfig. ## Fix Add derive_test_files_for() in scripts/mutmut.sh that maps each bin/wl_*.py module to the test files that actually exercise it. When MUTATE_PATH is set but TEST_RUNNER_FILES is not, auto-derive the right test list. Hard-fail with a clear error on unknown modules — silent fallback is exactly what produced the incident, so loud failure is the policy now. Special cases: - bin/wl_handler.py is explicitly REJECTED (integration tests only, multi-day runs; policy already documented but not mechanical before this commit) - Unknown MUTATE_PATH hard-fails with a pointer to: (a) add a case branch, OR (b) set TEST_RUNNER_FILES explicitly to override Escape hatch preserved: explicit TEST_RUNNER_FILES env var still bypasses derivation. The default invocation (no env vars) is fully backward-compatible — same module, same test list. ## New - 'mappings' subcommand prints the module -> tests table so contributors don't have to read derive_test_files_for to know what's mapped: scripts/mutmut.sh mappings ## Verification Smoke-tested all 5 derivation branches: - known module (bin/wl_csv.py) -> correct test list - handler -> rejection with policy message - unknown module -> rejection with mapping pointer - fim_watch (group entry) -> shared test file - default (no env vars) -> backward-compatible result Exit code 1 on all rejection paths (confirmed via separate run). ## Doc update docs/MUTATION_TESTING.md "Recommended improvements" section marks item 1 (the mapping table) CLOSED, renumbers the open items 1-4. Lessons-captured note unchanged. doc-drift: OK (33 docs against build 660).
1 parent eddcb62 commit d40304d

2 files changed

Lines changed: 191 additions & 20 deletions

File tree

‎docs/MUTATION_TESTING.md‎

Lines changed: 16 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -196,35 +196,38 @@ queued as v1.1 maintenance work.
196196

197197
---
198198

199-
## Recommended improvements (queued, not in scope of 2026-05-18 commit)
199+
## Recommended improvements
200200

201-
These are documented for the next mutation-testing session, not
202-
required for v1.0.0-rc1:
201+
### CLOSED (landed in v1.1 prep, 2026-05-19)
203202

204-
1. **Wire the mapping table into `scripts/mutmut.sh`** so
205-
`MUTATE_PATH=bin/wl_csv.py` automatically uses the correct
206-
`TEST_RUNNER_FILES`. A simple case statement based on the
207-
`MUTATE_PATH` value would prevent the misconfiguration that
208-
produced the discarded wl_csv survivors above. Keep the env
209-
override as escape hatch.
203+
1. ~~**Wire the mapping table into `scripts/mutmut.sh`**~~ — DONE.
204+
The script now auto-derives `TEST_RUNNER_FILES` from `MUTATE_PATH`
205+
via the `derive_test_files_for` function, and hard-fails on
206+
unknown modules (rather than silently falling back to wl_validation
207+
tests). Run `scripts/mutmut.sh mappings` to see the table.
208+
`bin/wl_handler.py` is explicitly rejected with a pointer to this
209+
doc. Escape hatch: explicit `TEST_RUNNER_FILES` env var still
210+
overrides auto-derivation.
210211

211-
2. **Switch the volume mount to read-only.** Replace
212+
### Open (queued for v1.1 release prep)
213+
214+
1. **Switch the volume mount to read-only.** Replace
212215
`-v "$REPO_ROOT:/work"` with `-v "$REPO_ROOT:/work:ro"` and have
213216
mutmut use a tmpfs (or container-local) scratch dir for its
214217
mutations. This removes the host-tree-corruption hazard entirely.
215218
Mutmut's internal cache lives in `mutants/.mutmut-cache` and can
216219
stay on a separate writable volume. Estimated effort: 1 hour
217220
(test the cache-survives-restart property).
218221

219-
3. **Re-run wl_validation with the correct selector** to get a
222+
2. **Re-run wl_validation with the correct selector** to get a
220223
fresh survivor count after the 2 new tests above. Expected
221224
result: ≤8 survivors (the equivalent mutants), down from 12.
222225

223-
4. **Run wl_csv.py with the correct selector**. Expected: real
226+
3. **Run wl_csv.py with the correct selector**. Expected: real
224227
survivor count, plausibly 5-20 genuine gaps in CSV diff /
225228
hash-registry logic.
226229

227-
5. **Add `bin/wl_audit.py` mutation pass.** The 37 reported survivors
230+
4. **Add `bin/wl_audit.py` mutation pass.** The 37 reported survivors
228231
on `wl_audit.py` from a prior session need the same re-validation
229232
under the correct selector before deciding what to do with them.
230233

‎scripts/mutmut.sh‎

Lines changed: 175 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,31 @@
1818
# scripts/mutmut.sh results # Show survivors
1919
# scripts/mutmut.sh show <id> # Show one survivor diff
2020
# scripts/mutmut.sh kill # Stop the cache container
21+
# scripts/mutmut.sh mappings # Print the module -> tests table
2122
#
2223
# Default <module> is bin/wl_validation.py (the security-critical
2324
# choke point that's most worth mutating). Other useful targets:
2425
# bin/wl_csv.py, bin/wl_versions.py, bin/wl_rbac.py, bin/wl_audit.py.
2526
#
27+
# Test selector auto-derivation
28+
# -----------------------------
29+
#
30+
# When you set MUTATE_PATH=<module> without also setting
31+
# TEST_RUNNER_FILES, the harness auto-selects the right test files
32+
# from the mapping table (see ``derive_test_files_for`` below).
33+
#
34+
# Origin: 2026-05-18 incident — a prior run used
35+
# MUTATE_PATH=bin/wl_csv.py with the unset TEST_RUNNER_FILES default,
36+
# which still pointed at wl_validation tests. The "survivors" reported
37+
# were all artifacts of test-selector / mutated-module mismatch (the
38+
# wl_validation tests never imported wl_csv at all, so EVERY csv
39+
# mutation trivially survived). See docs/MUTATION_TESTING.md.
40+
#
41+
# Mismatched config now hard-fails: if MUTATE_PATH is not in the
42+
# mapping table AND TEST_RUNNER_FILES is not explicitly set, the
43+
# script exits with a clear error rather than running garbage tests.
44+
# The escape hatch is to set TEST_RUNNER_FILES manually.
45+
#
2646
# Why a persistent container
2747
# --------------------------
2848
#
@@ -70,7 +90,110 @@ MUTATE_PATH="${MUTATE_PATH:-bin/wl_validation.py}"
7090
# baseline and block all mutations. Per-module scoping is also ~5x
7191
# faster: ~2-5s per mutation instead of ~15-20s.
7292
TEST_DIR="${TEST_DIR:-tests/unit}"
73-
TEST_RUNNER_FILES="${TEST_RUNNER_FILES:-tests/unit/test_validation.py tests/unit/test_ascii_validation.py tests/unit/test_validator_fuzz.py}"
93+
94+
# ── Module → test files mapping ──────────────────────────────────────
95+
#
96+
# Each mutated module maps to the test file(s) that actually
97+
# exercise it. When MUTATE_PATH is set but TEST_RUNNER_FILES is
98+
# unset, the script auto-derives the right test list. When neither
99+
# is set we fall through to the default bin/wl_validation.py target.
100+
#
101+
# To add a new module: add a case branch below + a row in
102+
# the ``mappings`` subcommand output. Update both together so the
103+
# help text never drifts from the actual logic.
104+
#
105+
# Special case: bin/wl_handler.py is NOT in the mapping. Mutating
106+
# it requires integration tests (~30s each × hundreds of mutations
107+
# = days). The script rejects MUTATE_PATH=bin/wl_handler.py with a
108+
# clear error pointing at this constraint.
109+
derive_test_files_for() {
110+
local mutate_path="$1"
111+
case "$mutate_path" in
112+
bin/wl_handler.py)
113+
echo "ERROR: bin/wl_handler.py mutation is forbidden by policy." >&2
114+
echo " Tests are integration-only (live Splunk required), ~30s per" >&2
115+
echo " invocation. Multiplied by hundreds of mutations = multi-day" >&2
116+
echo " runs. Mutate the handler's delegates (wl_validation, wl_csv," >&2
117+
echo " wl_rbac, etc.) instead — they cover the real logic." >&2
118+
echo " See scripts/mutmut.sh header + docs/MUTATION_TESTING.md." >&2
119+
return 1
120+
;;
121+
bin/wl_validation.py)
122+
echo "tests/unit/test_validation.py tests/unit/test_ascii_validation.py tests/unit/test_validator_fuzz.py"
123+
;;
124+
bin/wl_csv.py)
125+
echo "tests/unit/test_csv.py tests/unit/test_diff_fuzz.py"
126+
;;
127+
bin/wl_audit.py)
128+
echo "tests/unit/test_audit.py tests/unit/test_view_audit_dedup.py"
129+
;;
130+
bin/wl_approval.py)
131+
echo "tests/unit/test_approval.py tests/unit/test_approval_queue_state_machine.py tests/unit/test_pending_info_projection.py"
132+
;;
133+
bin/wl_rbac.py)
134+
echo "tests/unit/test_rbac.py"
135+
;;
136+
bin/wl_versions.py)
137+
echo "tests/unit/test_versions.py"
138+
;;
139+
bin/wl_limits.py)
140+
echo "tests/unit/test_limits.py"
141+
;;
142+
bin/wl_constants.py)
143+
echo "tests/unit/test_constants.py"
144+
;;
145+
bin/wl_filelock.py)
146+
echo "tests/unit/test_filelock.py"
147+
;;
148+
bin/wl_fim.py|bin/wl_fim_common.py|bin/wl_fim_watch.py)
149+
echo "tests/unit/test_fim_append_only.py"
150+
;;
151+
bin/wl_hmac_key.py)
152+
echo "tests/unit/test_hmac_sig_fuzz.py"
153+
;;
154+
bin/wl_logging.py)
155+
echo "tests/unit/test_logging.py"
156+
;;
157+
bin/wl_notify.py)
158+
echo "tests/unit/test_notify.py"
159+
;;
160+
bin/wl_presence.py)
161+
echo "tests/unit/test_presence.py"
162+
;;
163+
bin/wl_ratelimit.py)
164+
echo "tests/unit/test_ratelimit.py"
165+
;;
166+
bin/wl_replay.py)
167+
echo "tests/unit/test_replay.py"
168+
;;
169+
bin/wl_rules.py)
170+
echo "tests/unit/test_rules.py"
171+
;;
172+
bin/wl_trash.py)
173+
echo "tests/unit/test_trash.py"
174+
;;
175+
*)
176+
echo "ERROR: no test-file mapping for MUTATE_PATH='$mutate_path'." >&2
177+
echo " Either add a case branch in scripts/mutmut.sh :: derive_test_files_for," >&2
178+
echo " or set TEST_RUNNER_FILES explicitly to override the auto-derivation." >&2
179+
echo " Running mutmut without a matching test selector produces phantom" >&2
180+
echo " survivors (see docs/MUTATION_TESTING.md for the 2026-05-18 incident)." >&2
181+
echo "" >&2
182+
echo " Known mappings: scripts/mutmut.sh mappings" >&2
183+
return 1
184+
;;
185+
esac
186+
}
187+
188+
# Auto-derive TEST_RUNNER_FILES from MUTATE_PATH unless explicitly set.
189+
# Use `${VAR:+set}` semantics: if user passed a non-empty
190+
# TEST_RUNNER_FILES env var, honor it. Otherwise derive.
191+
if [ -z "${TEST_RUNNER_FILES:-}" ]; then
192+
if ! TEST_RUNNER_FILES=$(derive_test_files_for "$MUTATE_PATH"); then
193+
exit 1
194+
fi
195+
fi
196+
74197
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
75198

76199
# Posix path conversion for Windows Git Bash
@@ -148,17 +271,62 @@ cmd_kill() {
148271
fi
149272
}
150273

274+
cmd_mappings() {
275+
# Print the module-to-test mapping table. Kept hand-maintained
276+
# to mirror derive_test_files_for above. If you add a module
277+
# there, add a row here too.
278+
cat <<'EOF'
279+
Module → test files mapping (used when MUTATE_PATH is set but
280+
TEST_RUNNER_FILES is not):
281+
282+
bin/wl_validation.py tests/unit/test_validation.py
283+
tests/unit/test_ascii_validation.py
284+
tests/unit/test_validator_fuzz.py
285+
bin/wl_csv.py tests/unit/test_csv.py
286+
tests/unit/test_diff_fuzz.py
287+
bin/wl_audit.py tests/unit/test_audit.py
288+
tests/unit/test_view_audit_dedup.py
289+
bin/wl_approval.py tests/unit/test_approval.py
290+
tests/unit/test_approval_queue_state_machine.py
291+
tests/unit/test_pending_info_projection.py
292+
bin/wl_rbac.py tests/unit/test_rbac.py
293+
bin/wl_versions.py tests/unit/test_versions.py
294+
bin/wl_limits.py tests/unit/test_limits.py
295+
bin/wl_constants.py tests/unit/test_constants.py
296+
bin/wl_filelock.py tests/unit/test_filelock.py
297+
bin/wl_fim.py tests/unit/test_fim_append_only.py
298+
bin/wl_fim_common.py tests/unit/test_fim_append_only.py
299+
bin/wl_fim_watch.py tests/unit/test_fim_append_only.py
300+
bin/wl_hmac_key.py tests/unit/test_hmac_sig_fuzz.py
301+
bin/wl_logging.py tests/unit/test_logging.py
302+
bin/wl_notify.py tests/unit/test_notify.py
303+
bin/wl_presence.py tests/unit/test_presence.py
304+
bin/wl_ratelimit.py tests/unit/test_ratelimit.py
305+
bin/wl_replay.py tests/unit/test_replay.py
306+
bin/wl_rules.py tests/unit/test_rules.py
307+
bin/wl_trash.py tests/unit/test_trash.py
308+
309+
FORBIDDEN:
310+
bin/wl_handler.py (integration tests only — multi-day runs)
311+
312+
To override: set TEST_RUNNER_FILES env var explicitly.
313+
EOF
314+
}
315+
151316
cmd="${1:-run}"
152317
shift || true
153318

154319
case "$cmd" in
155-
run) cmd_run "$@" ;;
156-
results) cmd_results "$@" ;;
157-
show) cmd_show "$@" ;;
158-
kill) cmd_kill "$@" ;;
320+
run) cmd_run "$@" ;;
321+
results) cmd_results "$@" ;;
322+
show) cmd_show "$@" ;;
323+
kill) cmd_kill "$@" ;;
324+
mappings) cmd_mappings "$@" ;;
159325
*)
160-
echo "usage: scripts/mutmut.sh {run|results|show <id>|kill}" >&2
161-
echo " MUTATE_PATH=<file> TEST_PATH=<dir> overrides defaults" >&2
326+
echo "usage: scripts/mutmut.sh {run|results|show <id>|kill|mappings}" >&2
327+
echo " MUTATE_PATH=<file> select module (auto-derives tests)" >&2
328+
echo " TEST_RUNNER_FILES='<files>' explicit test list (overrides auto)" >&2
329+
echo " TEST_DIR=<dir> mutmut --tests-dir override" >&2
162330
exit 1
163331
;;
164332
esac

0 commit comments

Comments
 (0)