Skip to content

Commit c8aefba

Browse files
Copilotdalito
Copilot
and
dalito
committed
Add comprehensive testing plan for PR checks workflow
Co-authored-by: dalito <2648874+dalito@users.noreply.github.com>
1 parent 8e38d7e commit c8aefba

1 file changed

Lines changed: 212 additions & 0 deletions

File tree

‎.github/workflows/TESTING-pr-checks.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
# Testing Plan for PR Checks Workflow
2+
3+
This document outlines how to test the automated PR checks workflow to ensure it works correctly.
4+
5+
## Prerequisites
6+
7+
The workflow `.github/workflows/pr-checks.yml` will run automatically when:
8+
- A pull request is opened to the `main` branch
9+
- A pull request is synchronized (new commits pushed)
10+
- A pull request is reopened
11+
12+
## Test Scenarios
13+
14+
### Test 1: PR from Main Branch of Fork
15+
16+
**Setup:**
17+
1. Fork the repository to a personal account
18+
2. Make changes directly on the `main` branch of the fork
19+
3. Create a PR from `fork:main` to `upstream:main`
20+
21+
**Expected Behavior:**
22+
- Workflow runs successfully
23+
- A comment is posted explaining why submitting from main branch is problematic
24+
- Comment includes instructions on how to use feature branches
25+
- Comment is friendly and doesn't block the PR
26+
27+
**How to Verify:**
28+
- Check that `check-pr-submission` job completes
29+
- Look for comment from github-actions bot
30+
- Confirm comment text matches template in workflow
31+
- Verify comment is only posted once (not duplicated on subsequent pushes)
32+
33+
### Test 2: PR from Feature Branch
34+
35+
**Setup:**
36+
1. Fork the repository
37+
2. Create a feature branch: `git checkout -b test-feature`
38+
3. Make changes and push to the feature branch
39+
4. Create a PR from `fork:test-feature` to `upstream:main`
40+
41+
**Expected Behavior:**
42+
- Workflow runs successfully
43+
- NO comment about main branch is posted
44+
- No errors or warnings
45+
46+
**How to Verify:**
47+
- Check that `check-pr-submission` job completes
48+
- Confirm no comment about main branch appears
49+
- Check job logs show correct detection
50+
51+
### Test 3: PR from Organization Account
52+
53+
**Setup:**
54+
1. Fork the repository to an organization account (if available)
55+
2. Create a PR from the organization's fork
56+
3. Submit the PR
57+
58+
**Expected Behavior:**
59+
- Workflow runs successfully
60+
- An informational comment is posted about organization accounts
61+
- Comment explains potential implications
62+
- Comment includes option to switch to personal account
63+
64+
**How to Verify:**
65+
- Check that `check-pr-submission` job completes
66+
- Look for comment about organization account
67+
- Verify comment is informational and non-blocking
68+
69+
### Test 4: New Concepts with Proper Classification
70+
71+
**Setup:**
72+
1. Add new concepts to the vocabulary Excel file
73+
2. Ensure each new concept has a proper `skos:broader` relationship
74+
3. Verify the chain eventually reaches a top concept
75+
4. Submit the PR
76+
77+
**Expected Behavior:**
78+
- Workflow runs successfully
79+
- `check-top-concepts` job completes without errors
80+
- NO comment about missing classification is posted
81+
- Workflow passes
82+
83+
**How to Verify:**
84+
- Check that both jobs complete successfully
85+
- Review job logs to see concepts were analyzed
86+
- Confirm "All new concepts are properly classified" message in logs
87+
88+
### Test 5: New Concepts WITHOUT Proper Classification
89+
90+
**Setup:**
91+
1. Add new concepts to the vocabulary Excel file
92+
2. Intentionally omit `skos:broader` relationships OR
93+
3. Add broader relationship that doesn't chain to a top concept
94+
4. Submit the PR
95+
96+
**Expected Behavior:**
97+
- Workflow runs
98+
- `check-top-concepts` job detects unclassified concepts
99+
- A comment is posted listing the unclassified concepts
100+
- Comment explains why classification is important
101+
- Comment provides guidance on how to fix
102+
103+
**How to Verify:**
104+
- Check that `check-top-concepts` job runs
105+
- Verify comment lists the unclassified concept URIs
106+
- Confirm comment includes helpful guidance
107+
- Check that comment is updated (not duplicated) if more commits are pushed
108+
109+
### Test 6: PR with No New Concepts (Modification Only)
110+
111+
**Setup:**
112+
1. Modify existing concepts (change definitions, add synonyms, etc.)
113+
2. Do NOT add new concepts
114+
3. Submit the PR
115+
116+
**Expected Behavior:**
117+
- Workflow runs successfully
118+
- `check-top-concepts` job completes
119+
- Logs show "No new concepts added in this PR"
120+
- No classification comments posted
121+
122+
**How to Verify:**
123+
- Check job logs for the expected message
124+
- Confirm no classification-related comments appear
125+
- Workflow completes successfully
126+
127+
### Test 7: Documentation-Only Changes
128+
129+
**Setup:**
130+
1. Make changes only to .md files or documentation
131+
2. Don't modify vocabulary files at all
132+
3. Submit the PR
133+
134+
**Expected Behavior:**
135+
- `check-pr-submission` job runs (checks branch regardless of changes)
136+
- `check-top-concepts` job runs but finds no vocabulary changes
137+
- No issues or comments about concepts
138+
- Workflow completes successfully
139+
140+
**How to Verify:**
141+
- Both jobs complete
142+
- Logs show no vocabulary files changed
143+
- No errors or unexpected behavior
144+
145+
## Monitoring and Debugging
146+
147+
### Where to Check Workflow Runs
148+
149+
1. Go to the repository's Actions tab
150+
2. Click on "PR Submission Checks" workflow
151+
3. Select a specific run to see job details
152+
4. Review logs for each job and step
153+
154+
### Common Issues and Solutions
155+
156+
**Issue:** Workflow doesn't trigger
157+
- **Solution:** Check that the PR targets the `main` branch
158+
- **Solution:** Verify workflow file is on the base branch (main)
159+
160+
**Issue:** Python script fails to parse Turtle files
161+
- **Solution:** Check that vocabulary files are valid Turtle format
162+
- **Solution:** Review error logs for parsing issues
163+
- **Solution:** Ensure rdflib is installed correctly
164+
165+
**Issue:** Comments are duplicated
166+
- **Solution:** Check the logic that searches for existing comments
167+
- **Solution:** Verify comment detection regex is correct
168+
169+
**Issue:** Workflow fails with permissions error
170+
- **Solution:** Verify `pull-requests: write` permission is set
171+
- **Solution:** Check that `pull_request_target` is used (not `pull_request`)
172+
173+
## Success Criteria
174+
175+
The workflow is considered successful when:
176+
177+
1. ✅ All test scenarios pass as expected
178+
2. ✅ Comments are friendly, helpful, and not spammy
179+
3. ✅ No false positives or false negatives in detection
180+
4. ✅ Workflow completes in reasonable time (< 5 minutes)
181+
5. ✅ No security issues with fork handling
182+
6. ✅ Contributors find the feedback helpful (gather feedback over time)
183+
184+
## Future Enhancements
185+
186+
Based on testing and user feedback, consider:
187+
188+
- Adding more vocabulary-specific checks
189+
- Improving detection accuracy
190+
- Adding links to specific sections of guidelines
191+
- Providing automated fixes where possible
192+
- Collecting metrics on common issues
193+
194+
## Rollback Plan
195+
196+
If the workflow causes issues:
197+
198+
1. Disable by adding `if: false` to the top of each job
199+
2. Push the change to main branch
200+
3. Investigate and fix the issue
201+
4. Re-enable and test on a fork first
202+
5. Merge fix when confirmed working
203+
204+
## Feedback Collection
205+
206+
To gather feedback on the automated checks:
207+
208+
1. Monitor PR comments and discussions
209+
2. Create a feedback issue for contributors to share experiences
210+
3. Review which comments are most helpful
211+
4. Adjust messaging based on contributor responses
212+
5. Track if issues decline over time (indicating effectiveness)

0 commit comments

Comments
 (0)