Commit 2ba9121
Improve verify-tests-fail-without-fix Skill (#33513)
<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!
## Description
This PR improves the `verify-tests-fail-without-fix` skill by fixing
fork repository detection, improving error handling, and making full
verification the recommended/default behavior.
## Changes
### 1. **Added `git fetch origin` for Accurate Diffs**
- Fetches the base branch from origin before computing git diff
- Ensures accurate fix file detection even in fork repositories
- Prevents false negatives when base branch is stale locally
### 2. **Improved Verification Mode Behavior**
- **Made full verification the primary/default mode** - tests must FAIL
without fix and PASS with fix
- **Added `-RequireFullVerification` parameter** - prevents silent
fallback to verify-only mode
- **Improved error handling** - script now warns/errors if no fix files
detected (instead of silently switching modes)
- **Verify-only mode still available** - runs when no fix files detected
AND `-RequireFullVerification` not set
- **Clearer expectations** - user explicitly controls whether
verify-only fallback is allowed
### 3. **Improved Documentation**
- Updated SKILL.md to reflect improved mode behavior
- Clarified requirements (fix files + test files for full verification)
- Updated mode selection table to show both modes clearly
- Added troubleshooting guidance for "no fix files detected" scenario
### 4. **Better Error Handling**
- Script exits early with actionable error if no fix files found AND
`-RequireFullVerification` set
- Provides explicit solutions (use `-FixFiles` or `-BaseBranch`
explicitly)
- Reduced ambiguity - user knows exactly what mode is running
## Why These Changes?
### Problem 1: Fork Repository Detection
When working in a fork, the base branch might not be up-to-date locally,
causing `git diff` to produce inaccurate results. This led to:
- Missing fix files in the diff
- False "only test files" mode activation
- Incorrect verification results
**Solution**: Always fetch the base branch from origin before computing
diff.
### Problem 2: Confusing Auto-Detection
The previous auto-detection system was unclear:
- Silent fallback to "verify failure only" when no fix files detected
- Users didn't know which mode was running
- Made debugging harder
**Solution**:
- Add `-RequireFullVerification` parameter to make expectations explicit
- Error immediately if user expects full verification but no fix files
found
- Maintain verify-only mode for legitimate test-first workflows
### Problem 3: Silent Failures
When no fix files were detected (due to stale base branch or wrong base
detection), the script would silently fall back to "verify failure only"
mode, which wasn't the intended behavior.
**Solution**: Fail fast with clear error message and troubleshooting
steps when `-RequireFullVerification` is set.
## Testing
Verified the changes work correctly with:
- ✅ Fork repositories (fetch ensures accurate base branch)
- ✅ PRs with fix files + test files (full verification runs)
- ✅ Clear error when no fix files detected with
`-RequireFullVerification`
- ✅ Verify-only mode still works when appropriate (without the flag)
- ✅ `-RequireFullVerification` parameter enforces strict mode
## Example Usage
```bash
# Recommended for PR validation - ensures full verification
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification
# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification
# Override fix files or base branch if auto-detection fails
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -FixFiles @("src/Core/File.cs") -BaseBranch "main"
# Test-first workflow (verify tests fail before writing fix)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios
```
## Breaking Changes
1 parent 2cb89dd commit 2ba9121
File tree
3 files changed
+461
-214
lines changed- .github
- agents
- skills/verify-tests-fail-without-fix
- scripts
3 files changed
+461
-214
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
407 | 407 | | |
408 | 408 | | |
409 | 409 | | |
410 | | - | |
| 410 | + | |
411 | 411 | | |
412 | 412 | | |
413 | 413 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
11 | 16 | | |
12 | 17 | | |
13 | | - | |
| 18 | + | |
14 | 19 | | |
15 | 20 | | |
16 | 21 | | |
17 | 22 | | |
18 | 23 | | |
19 | 24 | | |
20 | | - | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
21 | 34 | | |
22 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
23 | 38 | | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
| 39 | + | |
28 | 40 | | |
29 | | - | |
30 | | - | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
31 | 51 | | |
32 | 52 | | |
33 | 53 | | |
34 | | - | |
| 54 | + | |
35 | 55 | | |
36 | 56 | | |
37 | 57 | | |
38 | 58 | | |
39 | | - | |
40 | | - | |
| 59 | + | |
| 60 | + | |
41 | 61 | | |
42 | 62 | | |
43 | 63 | | |
44 | | - | |
| 64 | + | |
45 | 65 | | |
46 | 66 | | |
47 | | - | |
| 67 | + | |
48 | 68 | | |
49 | | - | |
| 69 | + | |
| 70 | + | |
50 | 71 | | |
51 | 72 | | |
52 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
53 | 92 | | |
54 | 93 | | |
55 | 94 | | |
56 | 95 | | |
| 96 | + | |
57 | 97 | | |
58 | | - | |
| 98 | + | |
59 | 99 | | |
60 | 100 | | |
61 | 101 | | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | 102 | | |
79 | 103 | | |
80 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
81 | 108 | | |
82 | 109 | | |
83 | 110 | | |
84 | 111 | | |
85 | 112 | | |
86 | 113 | | |
87 | | - | |
88 | | - | |
| 114 | + | |
| 115 | + | |
89 | 116 | | |
0 commit comments