Skip to content

Commit 4384c00

Browse files
authored
Copilot CI: Structured phase outputs, autonomous execution, iOS support, and CI pipeline (dotnet#34040)
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Description of Change Major overhaul of the Copilot CI PR reviewer agent infrastructure with five key areas: **1. Structured Phase Output Artifacts** - Replace monolithic state file (`pr-XXXXX.md`) with per-phase `content.md` files under `CustomAgentLogsTmp/PRState/{PRNumber}/PRAgent/{phase}/content.md` - `post-ai-summary-comment.ps1` now auto-loads from phase directories instead of a single state file - Removed `-Content` and `-StateFile` parameters; script always auto-discovers phase files - Updated all comment scripts to use new `PRAgent/` subdirectory paths **2. Autonomous (Non-Interactive) Execution** - Agent no longer stops and asks user on environment blockers — retries once, then skips and continues - Removed all "STOP and ask user" guidance from agent docs (`pr.md`, `SHARED-RULES.md`, `post-gate.md`, `PLAN-TEMPLATE.md`) - Added "AUTONOMOUS EXECUTION" section to `Review-PR.ps1` prompt - `verify-tests-fail.ps1` now errors if `-PRNumber` is not provided (no "unknown" fallback) **3. iOS Platform Support for CI Pipeline** - Added `Platform` parameter to `ci-copilot.yml` (android/ios) - Skip Android SDK/JDK/emulator provisioning when Platform=ios - Added iOS simulator boot step with DEVICE_UDID propagation - Install both Appium drivers (uiautomator2 + xcuitest) **4. CI Pipeline & Test Infrastructure** - New `eng/pipelines/ci-copilot.yml` pipeline definition (682 lines) - `BuildAndRunHostApp.ps1`: Set `APPIUM_LOG_FILE` env var, collect test screenshots/page source artifacts - `provision.yml`: Changed defaults for copilot CI builds - Deleted 4 archived `.github/agent-pr-session/` files **5. Documentation & Cleanup** - Removed all references to "state file" from agent docs, skills, and scripts - Updated `learn-from-pr`, `try-fix` skills to remove state file references - Fixed stdin hang and validation improvements in comment scripts ### Issues Fixed Infrastructure improvements — no specific issue number. ### ⚠️ Review Notes **`provision.yml` and `variables.yml` changes affect shared CI infrastructure** — these files are used by all pipelines, not just copilot CI. The changes should be carefully reviewed to ensure they do not break `maui-pr`, `dotnet-maui`, or other existing pipelines.
2 parents e4e3573 + 006a4de commit 4384c00

24 files changed

Lines changed: 1505 additions & 629 deletions

.github/agents/learn-from-pr.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,6 @@ Present a summary:
8989
| Situation | Action |
9090
|-----------|--------|
9191
| PR not found | Ask user to verify PR number |
92-
| No session markdown | Proceed with PR diff analysis only |
9392
| Target file doesn't exist | Create if instruction/architecture doc, skip if code |
9493
| Duplicate content exists | Skip, note in report |
9594
| Unclear where to add | Ask user for guidance |

.github/agents/pr.md

Lines changed: 16 additions & 169 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: pr
3-
description: Sequential 4-phase workflow for GitHub issues - Pre-Flight, Gate, Fix, Report. Phases MUST complete in order. State tracked in CustomAgentLogsTmp/PRState/
3+
description: Sequential 4-phase workflow for GitHub issues - Pre-Flight, Gate, Fix, Report. Phases MUST complete in order.
44
---
55

66
# .NET MAUI Pull Request Agent
@@ -48,13 +48,13 @@ After Gate passes, read `.github/agents/pr/post-gate.md` for **Phases 3-4**.
4848
- Follow Templates EXACTLY (no `open` attributes, no "improvements")
4949
- No Direct Git Commands (use `gh pr diff/view`, let scripts handle files)
5050
- Use Skills' Scripts (don't bypass with manual commands)
51-
- Stop on Environment Blockers (strict retry limits, report and ask user)
51+
- Stop on Environment Blockers (retry once, then skip and continue autonomously)
5252
- Multi-Model Configuration (5 models for Phase 4)
5353
- Platform Selection (must be affected AND available on host)
5454

5555
**Key points:**
5656
- ❌ Never run `git checkout`, `git switch`, `git stash`, `git reset` - agent is always on correct branch
57-
- ❌ Never continue after environment blocker - STOP and ask user
57+
- ❌ Never stop and ask user - use best judgment to skip blocked phases and continue
5858
- ❌ Never mark phase ✅ with [PENDING] fields remaining
5959

6060
Phase 3 uses a 5-model exploration workflow. See `post-gate.md` for detailed instructions after Gate passes.
@@ -65,8 +65,6 @@ Phase 3 uses a 5-model exploration workflow. See `post-gate.md` for detailed ins
6565

6666
> **⚠️ SCOPE**: Document only. No code analysis. No fix opinions. No running tests.
6767
68-
**🚨 CRITICAL: Create the state file BEFORE doing anything else.**
69-
7068
### ❌ Pre-Flight Boundaries (What NOT To Do)
7169

7270
| ❌ Do NOT | Why | When to do it |
@@ -79,141 +77,11 @@ Phase 3 uses a 5-model exploration workflow. See `post-gate.md` for detailed ins
7977

8078
### ✅ What TO Do in Pre-Flight
8179

82-
- Create/check state file
8380
- Read issue description and comments
8481
- Note platforms affected (from labels)
8582
- Identify files changed (if PR exists)
8683
- Document disagreements and edge cases from comments
8784

88-
### Step 0: Check for Existing State File or Create New One
89-
90-
**State file location**: `CustomAgentLogsTmp/PRState/pr-XXXXX.md`
91-
92-
**Naming convention:**
93-
- If starting from **PR #12345** → Name file `pr-12345.md` (use PR number)
94-
- If starting from **Issue #33356** (no PR yet) → Name file `pr-33356.md` (use issue number as placeholder)
95-
- When PR is created later → Rename to use actual PR number
96-
97-
```bash
98-
# Check if state file exists
99-
mkdir -p CustomAgentLogsTmp/PRState
100-
if [ -f "CustomAgentLogsTmp/PRState/pr-XXXXX.md" ]; then
101-
echo "State file exists - resuming session"
102-
cat CustomAgentLogsTmp/PRState/pr-XXXXX.md
103-
else
104-
echo "Creating new state file"
105-
fi
106-
```
107-
108-
**If the file EXISTS**: Read it to determine your current phase and resume from there. Look for:
109-
- Which phase has `▶️ IN PROGRESS` status - that's where you left off
110-
- Which phases have `✅ PASSED` status - those are complete
111-
- Which phases have `⏳ PENDING` status - those haven't started
112-
113-
**If the file does NOT exist**: Create it with the template structure:
114-
115-
```markdown
116-
# PR Review: #XXXXX - [Issue Title TBD]
117-
118-
**Date:** [TODAY] | **Issue:** [#XXXXX](https://github.com/dotnet/maui/issues/XXXXX) | **PR:** [#YYYYY](https://github.com/dotnet/maui/pull/YYYYY) or None
119-
120-
## ⏳ Status: IN PROGRESS
121-
122-
| Phase | Status |
123-
|-------|--------|
124-
| Pre-Flight | ▶️ IN PROGRESS |
125-
| 🚦 Gate | ⏳ PENDING |
126-
| 🔧 Fix | ⏳ PENDING |
127-
| 📋 Report | ⏳ PENDING |
128-
129-
---
130-
131-
<details>
132-
<summary><strong>📋 Issue Summary</strong></summary>
133-
134-
[From issue body]
135-
136-
**Steps to Reproduce:**
137-
1. [Step 1]
138-
2. [Step 2]
139-
140-
**Platforms Affected:**
141-
- [ ] iOS
142-
- [ ] Android
143-
- [ ] Windows
144-
- [ ] MacCatalyst
145-
146-
</details>
147-
148-
<details>
149-
<summary><strong>📁 Files Changed</strong></summary>
150-
151-
| File | Type | Changes |
152-
|------|------|---------|
153-
| `path/to/fix.cs` | Fix | +X lines |
154-
| `path/to/test.cs` | Test | +Y lines |
155-
156-
</details>
157-
158-
<details>
159-
<summary><strong>💬 PR Discussion Summary</strong></summary>
160-
161-
**Key Comments:**
162-
- [Notable comments from issue/PR discussion]
163-
164-
**Reviewer Feedback:**
165-
- [Key points from review comments]
166-
167-
**Disagreements to Investigate:**
168-
| File:Line | Reviewer Says | Author Says | Status |
169-
|-----------|---------------|-------------|--------|
170-
171-
**Author Uncertainty:**
172-
- [Areas where author expressed doubt]
173-
174-
</details>
175-
176-
<details>
177-
<summary><strong>🚦 Gate - Test Verification</strong></summary>
178-
179-
**Status**: ⏳ PENDING
180-
181-
- [ ] Tests FAIL (bug reproduced)
182-
183-
**Result:** [PENDING]
184-
185-
</details>
186-
187-
<details>
188-
<summary><strong>🔧 Fix Candidates</strong></summary>
189-
190-
**Status**: ⏳ PENDING
191-
192-
| # | Source | Approach | Test Result | Files Changed | Notes |
193-
|---|--------|----------|-------------|---------------|-------|
194-
| PR | PR #XXXXX | [PR's approach - from Pre-Flight] | ⏳ PENDING (Gate) | [files] | Original PR - validated by Gate |
195-
196-
**Note:** try-fix candidates (1, 2, 3...) are added during Phase 3. PR's fix is reference only.
197-
198-
**Exhausted:** No
199-
**Selected Fix:** [PENDING]
200-
201-
</details>
202-
203-
---
204-
205-
**Next Step:** After Gate passes, read `.github/agents/pr/post-gate.md` and continue with phases 3-4.
206-
```
207-
208-
This file:
209-
- Serves as your TODO list for all phases
210-
- Tracks progress if interrupted
211-
- Must exist before you start gathering context
212-
- **Always include when saving changes** (to `CustomAgentLogsTmp/PRState/`)
213-
- **Phases 3-4 sections are added AFTER Gate passes** (see `pr/post-gate.md`)
214-
215-
**Then gather context and update the file as you go.**
216-
21785
### Step 1: Gather Context (depends on starting point)
21886

21987
**If starting from a PR:**
@@ -258,11 +126,9 @@ gh pr view XXXXX --json comments --jq '.comments[] | select(.body | contains("Fi
258126
- Contains structured analysis (Root Cause, Platform Comparison, etc.)
259127

260128
**If prior agent review found:**
261-
1. **Extract and use as state file content** - The review IS the completed state
262-
2. Parse the phase statuses to determine what's already done
263-
3. Import all findings (fix candidates, test results)
264-
4. Update your local state file with this content
265-
5. Resume from whichever phase is not yet complete (or report as done)
129+
1. Parse the phase statuses to determine what's already done
130+
2. Import all findings (fix candidates, test results)
131+
3. Resume from whichever phase is not yet complete (or report as done)
266132

267133
**Do NOT:**
268134
- Start from scratch if a complete review already exists
@@ -271,8 +137,6 @@ gh pr view XXXXX --json comments --jq '.comments[] | select(.body | contains("Fi
271137

272138
### Step 3: Document Key Findings
273139

274-
Update the state file `CustomAgentLogsTmp/PRState/pr-XXXXX.md`:
275-
276140
**If PR exists** - Document disagreements and reviewer feedback:
277141
| File:Line | Reviewer Says | Author Says | Status |
278142
|-----------|---------------|-------------|--------|
@@ -308,21 +172,12 @@ The test result will be updated to `✅ PASS (Gate)` after Gate passes.
308172

309173
### Step 5: Complete Pre-Flight
310174

311-
**🚨 MANDATORY: Update state file**
312-
313-
**Update state file** - Change Pre-Flight status and populate with gathered context:
314-
1. Change Pre-Flight status from `▶️ IN PROGRESS` to `✅ COMPLETE`
315-
2. Fill in issue summary, platforms affected, regression info
316-
3. Add edge cases and any disagreements (if PR exists)
317-
4. Change 🚦 Gate status to `▶️ IN PROGRESS`
318-
319-
**Before marking ✅ COMPLETE, verify state file contains:**
320-
- [ ] Issue summary filled (not [PENDING])
321-
- [ ] Platform checkboxes marked
322-
- [ ] Files Changed table populated (if PR exists)
323-
- [ ] PR Discussion Summary documented (if PR exists)
324-
- [ ] All [PENDING] placeholders replaced
325-
- [ ] State file saved
175+
Verify the following before proceeding:
176+
- [ ] Issue summary captured
177+
- [ ] Platform information noted
178+
- [ ] Files changed identified (if PR exists)
179+
- [ ] PR discussion summarized (if PR exists)
180+
- [ ] **Write phase output to `CustomAgentLogsTmp/PRState/{PRNumber}/PRAgent/pre-flight/content.md`** (see SHARED-RULES.md "Phase Output Artifacts")
326181

327182
---
328183

@@ -356,7 +211,7 @@ find src/Controls/tests -name "*XXXXX*" -type f 2>/dev/null
356211
**🚨 CRITICAL: Choose a platform that is BOTH affected by the bug AND available on the current host.**
357212

358213
**Identify affected platforms** from Pre-Flight:
359-
- Check the "Platforms Affected" checkboxes in the state file
214+
- Check the platforms affected from Pre-Flight context
360215
- Check issue labels (e.g., `platform/iOS`, `platform/Android`)
361216
- Check which platform-specific files the PR modifies
362217

@@ -416,18 +271,11 @@ See `.github/skills/verify-tests-fail-without-fix/SKILL.md` for full skill docum
416271

417272
### Complete 🚦 Gate
418273

419-
**🚨 MANDATORY: Update state file**
420-
421-
**Update state file**:
422-
1. Fill in **Result**: `PASSED ✅`
423-
2. Change 🚦 Gate status to `✅ PASSED`
424-
3. Proceed to Phase 3
425-
426-
**Before marking ✅ PASSED, verify state file contains:**
427-
- [ ] Result shows PASSED ✅ or FAILED ❌
274+
Verify the following before proceeding:
275+
- [ ] Test result documented (PASSED ✅ or FAILED ❌)
428276
- [ ] Test behavior documented
429277
- [ ] Platform tested noted
430-
- [ ] State file saved
278+
- [ ] **Write phase output to `CustomAgentLogsTmp/PRState/{PRNumber}/PRAgent/gate/content.md`** (see SHARED-RULES.md "Phase Output Artifacts")
431279

432280
---
433281

@@ -445,7 +293,6 @@ See `.github/skills/verify-tests-fail-without-fix/SKILL.md` for full skill docum
445293
-**Looking at implementation code during Pre-Flight** - Just gather issue/PR context
446294
-**Forming opinions on the fix during Pre-Flight** - That's Phase 3
447295
-**Running tests during Pre-Flight** - That's Phase 2 (Gate)
448-
-**Not creating state file first** - ALWAYS create state file before gathering context
449296
-**Skipping to Phase 3** - Gate MUST pass first
450297

451298
## Common Gate Mistakes

.github/agents/pr/PLAN-TEMPLATE.md

Lines changed: 15 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
## 🚨 Critical Rules (Summary)
1313

1414
See `SHARED-RULES.md` for complete details. Key points:
15-
- **Environment Blockers**: STOP immediately, report, ask user (strict retry limits)
15+
- **Environment Blockers**: Skip blocked phase and continue autonomously (no human operator)
1616
- **No Git Commands**: Never checkout/switch branches - agent is always on correct branch
1717
- **Gate via Task Agent**: Never run inline (prevents fabrication)
1818
- **Multi-Model try-fix**: 5 models, SEQUENTIAL only
@@ -23,16 +23,14 @@ See `SHARED-RULES.md` for complete details. Key points:
2323
## Work Plan
2424

2525
### Phase 1: Pre-Flight
26-
- [ ] Create state file: `CustomAgentLogsTmp/PRState/pr-XXXXX.md`
2726
- [ ] Gather PR metadata (title, body, labels, author)
2827
- [ ] Fetch and read linked issue
2928
- [ ] Fetch PR comments and review feedback
3029
- [ ] Check for prior agent reviews (import and resume if found)
3130
- [ ] Document platforms affected
3231
- [ ] Classify changed files (fix vs test)
3332
- [ ] Document PR's fix approach in Fix Candidates table
34-
- [ ] Update state file: Pre-Flight → ✅ COMPLETE
35-
- [ ] Save state file
33+
- [ ] **Write `PRAgent/pre-flight/content.md`**
3634

3735
**Boundaries:** No code analysis, no fix opinions, no test running
3836

@@ -46,11 +44,10 @@ See `SHARED-RULES.md` for complete details. Key points:
4644
"Run verify-tests-fail-without-fix skill
4745
Platform: [X], TestFilter: 'IssueXXXXX', RequireFullVerification: true"
4846
```
49-
- [ ] ⛔ If environment blocker: STOP, report, ask user
47+
- [ ] ⛔ If environment blocker: retry once, then skip and document
5048
- [ ] Verify: Tests FAIL without fix, PASS with fix
5149
- [ ] If Gate fails: STOP, request test fixes
52-
- [ ] Update state file: Gate → ✅ PASSED
53-
- [ ] Save state file
50+
- [ ] **Write `PRAgent/gate/content.md`**
5451

5552
### Phase 3: Fix 🔧
5653
*(Only if Gate ✅ PASSED)*
@@ -61,7 +58,7 @@ See `SHARED-RULES.md` for complete details. Key points:
6158
- [ ] gpt-5.2
6259
- [ ] gpt-5.3-codex
6360
- [ ] gemini-3-pro-preview
64-
- [ ] ⛔ If blocker: STOP, report, ask user
61+
- [ ] ⛔ If blocker: retry once, skip remaining models, proceed to Report
6562
- [ ] Record: approach, result, files, failure analysis
6663

6764
**Round 2+: Cross-Pollination (MANDATORY)**
@@ -75,37 +72,33 @@ See `SHARED-RULES.md` for complete details. Key points:
7572
- [ ] Mark Exhausted: Yes
7673
- [ ] Compare passing candidates with PR's fix
7774
- [ ] Select best fix (results → simplicity → robustness)
78-
- [ ] Update state file: Fix → ✅ COMPLETE
79-
- [ ] Save state file
75+
- [ ] **Write `PRAgent/try-fix/content.md`**
8076

8177
### Phase 4: Report 📋
8278
*(Only if Phases 1-3 complete)*
8379

8480
- [ ] Run `pr-finalize` skill
8581
- [ ] Generate review: root cause, candidates, recommendation
86-
- [ ] Post AI Summary comment (PR phases + try-fix):
82+
- [ ] **Write `PRAgent/report/content.md`**
83+
- [ ] Post AI Summary comment (auto-loads from PRAgent/*/content.md):
8784
```bash
88-
pwsh .github/skills/ai-summary-comment/scripts/post-ai-summary-comment.ps1 -PRNumber XXXXX -SkipValidation
85+
pwsh .github/skills/ai-summary-comment/scripts/post-ai-summary-comment.ps1 -PRNumber XXXXX
8986
pwsh .github/skills/ai-summary-comment/scripts/post-try-fix-comment.ps1 -IssueNumber XXXXX
9087
```
9188
- [ ] Post PR Finalization comment (separate):
9289
```bash
93-
pwsh .github/skills/ai-summary-comment/scripts/post-pr-finalize-comment.ps1 -PRNumber XXXXX -SummaryFile CustomAgentLogsTmp/PRState/pr-XXXXX.md
90+
pwsh .github/skills/ai-summary-comment/scripts/post-pr-finalize-comment.ps1 -PRNumber XXXXX -SummaryFile CustomAgentLogsTmp/PRState/XXXXX/PRAgent/pr-finalize/pr-finalize-summary.md
9491
```
95-
- [ ] Update state file: Report → ✅ COMPLETE
96-
- [ ] Save final state file
9792

9893
---
9994

10095
## Quick Reference
10196

10297
| Phase | Key Action | Blocker Response |
10398
|-------|------------|------------------|
104-
| Pre-Flight | Create state file | N/A |
105-
| Gate | Task agent → verify script | ⛔ STOP, report, ask |
106-
| Fix | Multi-model try-fix | ⛔ STOP, report, ask |
107-
| Report | Post via skill | ⛔ STOP, report, ask |
99+
| Pre-Flight | Gather context | N/A |
100+
| Gate | Task agent → verify script | Skip, report incomplete |
101+
| Fix | Multi-model try-fix | Skip remaining, proceed to Report |
102+
| Report | Post via skill | Document what completed |
108103

109-
**State file:** `CustomAgentLogsTmp/PRState/pr-XXXXX.md`
110-
111-
**Never:** Mark BLOCKED and continue, claim success without tests, bypass scripts
104+
**Never:** Claim success without tests, bypass scripts, stop and ask user

0 commit comments

Comments
 (0)