Skip to content

Commit 720a9d4

Browse files
PureWeengithub-actions[bot]Copilot
authored
Allow fork PRs to auto-trigger evaluate-pr-tests workflow (dotnet#34655)
## Summary Enables the copilot-evaluate-tests gh-aw workflow to run on fork PRs by adding `forks: ["*"]` to the `pull_request` trigger and removing the fork guard from `Checkout-GhAwPr.ps1`. ## Changes 1. **copilot-evaluate-tests.md**: Added `forks: ["*"]` to opt out of gh-aw auto-injected fork activation guard. Scoped `Checkout-GhAwPr.ps1` step to `workflow_dispatch` only (redundant for other triggers since platform handles checkout). 2. **copilot-evaluate-tests.lock.yml**: Recompiled via `gh aw compile` — fork guard removed from activation `if:` conditions. 3. **Checkout-GhAwPr.ps1**: Removed the `isCrossRepository` fork guard. Updated header docs and restore comments to accurately describe behavior for all trigger×fork combinations (including corrected step ordering). 4. **gh-aw-workflows.instructions.md**: Updated all stale references to the removed fork guard. Documented `forks: ["*"]` opt-in, clarified residual risk model for fork PRs, and updated troubleshooting table. ## Security Model Fork PRs are safe because: - Agent runs in **sandboxed container** with all credentials scrubbed - Output limited to **1 comment** via `safe-outputs: add-comment: max: 1` - Agent **prompt comes from base branch** (`runtime-import`) — forks cannot alter instructions - Pre-flight check catches missing `SKILL.md` if fork isn't rebased on `main` - No workspace code is executed with `GITHUB_TOKEN` (checkout without execution) ## Testing - ✅ `workflow_dispatch` tested against fork PR dotnet#34621 - ✅ Lock.yml statically verified — fork guard removed from `if:` conditions - ⏳ `pull_request` trigger on fork PRs can only be verified post-merge (GitHub Actions reads lock.yml from default branch) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 6717d1c commit 720a9d4

File tree

4 files changed

+44
-52
lines changed

4 files changed

+44
-52
lines changed

.github/instructions/gh-aw-workflows.instructions.md

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ The prompt is built in the **activation job** via `{{#runtime-import .github/wor
4444

4545
### Fork PR Activation Gate
4646

47-
`gh aw compile` automatically injects a fork guard into the activation job's `if:` condition: `head.repo.id == repository_id`. This blocks fork PRs on `pull_request` events. This is **platform behavior** — do not add it manually.
47+
By default, `gh aw compile` automatically injects a fork guard into the activation job's `if:` condition: `head.repo.id == repository_id`. This blocks fork PRs on `pull_request` events.
48+
49+
To **allow fork PRs**, add `forks: ["*"]` to the `pull_request` trigger in the `.md` frontmatter. The compiler removes the auto-injected guard from the compiled `if:` conditions. This is safe when the workflow uses the `Checkout-GhAwPr.ps1` pattern (checkout + trusted-infra restore) and the agent is sandboxed.
4850

4951
## Fork PR Handling
5052

@@ -58,16 +60,17 @@ Reference: https://securitylab.github.com/resources/github-actions-preventing-pw
5860

5961
| Trigger | `checkout_pr_branch.cjs` runs? | Fork handling |
6062
|---------|-------------------------------|---------------|
61-
| `pull_request` | ✅ Yes | Blocked by auto-generated activation gate |
63+
| `pull_request` (default) | ✅ Yes | Blocked by auto-generated activation gate unless `forks: ["*"]` is set |
64+
| `pull_request` + `forks: ["*"]` | ✅ Yes | ✅ Works — user steps restore trusted infra before agent runs |
6265
| `workflow_dispatch` | ❌ Skipped | ✅ Works — user steps handle checkout and restore is final |
6366
| `issue_comment` (same-repo) | ✅ Yes | ✅ Works — files already on PR branch |
64-
| `issue_comment` (fork) | N/A | ❌ Blocked by fail-closed fork guard in `Checkout-GhAwPr.ps1` |
67+
| `issue_comment` (fork) | ✅ Yes | ⚠️ Works — `checkout_pr_branch.cjs` re-checks out fork branch after user steps, potentially overwriting restored infra. Acceptable because agent is sandboxed (no credentials, max 1 comment via safe-outputs). Pre-flight check catches missing `SKILL.md` if fork isn't rebased. |
6568

6669
### The `issue_comment` + Fork Problem
6770

68-
For `/slash-command` triggers on fork PRs, `checkout_pr_branch.cjs` runs AFTER all user steps and re-checks out the fork branch. This overwrites any files restored by user steps (e.g., `.github/skills/`). There is no way to run user steps after platform steps. A fork could include a crafted `SKILL.md` that alters the agent's evaluation behavior.
71+
For `/slash-command` triggers on fork PRs, `checkout_pr_branch.cjs` runs AFTER all user steps and re-checks out the fork branch. This overwrites any files restored by user steps (e.g., `.github/skills/`). A fork could include a crafted `SKILL.md` that alters the agent's evaluation behavior.
6972

70-
**Current approach (fail-closed fork guard):** `Checkout-GhAwPr.ps1` checks `isCrossRepository` via `gh pr view` for `issue_comment` triggers. If the PR is from a fork or the API call fails, the script exits with code 1. Fork PRs should use `workflow_dispatch` instead, where `checkout_pr_branch.cjs` is skipped and the user step restore is the final workspace state.
73+
**Accepted residual risk:** The agent runs in a sandboxed container with all credentials scrubbed. The worst outcome is a manipulated evaluation comment (`safe-outputs: add-comment: max: 1`). The agent has no ability to push code, access secrets, or exfiltrate data. The pre-flight check in the agent prompt catches the case where `SKILL.md` is missing entirely (fork not rebased on `main`).
7174

7275
**Upstream issue:** [github/gh-aw#18481](https://github.com/github/gh-aw/issues/18481) — "Using gh-aw in forks of repositories"
7376

@@ -86,16 +89,16 @@ steps:
8689
8790
The script:
8891
1. Captures the base branch SHA before checkout
89-
2. **Fork guard** (`issue_comment` only): checks `isCrossRepository` — exits 1 if fork or API failure
90-
3. Checks out the PR branch via `gh pr checkout`
91-
4. Deletes `.github/skills/` and `.github/instructions/` (prevents fork-added files)
92-
5. Restores them from the base branch SHA (best-effort, non-fatal)
92+
2. Checks out the PR branch via `gh pr checkout`
93+
3. Deletes `.github/skills/` and `.github/instructions/` (prevents fork-added files)
94+
4. Restores them from the base branch SHA (best-effort, non-fatal)
9395

9496
**Behavior by trigger:**
95-
- **`workflow_dispatch`**: Fork guard skipped. Platform checkout is skipped, so the restore IS the final workspace state (trusted files from base branch)
96-
- **`issue_comment`** (same-repo): Fork guard passes. Platform re-checks out PR branch — files already match, effectively a no-op
97-
- **`issue_comment`** (fork): Fork guard rejects — exits 1 with actionable notice to use `workflow_dispatch`
98-
- **`pull_request`** (same-repo): Fork guard skipped. Files already exist, restore is a no-op
97+
- **`workflow_dispatch`**: Platform checkout is skipped, so the restore IS the final workspace state (trusted files from base branch)
98+
- **`pull_request`** (same-repo): User step restores trusted infra. `checkout_pr_branch.cjs` runs after and re-checks out PR branch — for same-repo PRs, skill files typically match main unless the PR modified them.
99+
- **`pull_request`** (fork with `forks: ["*"]`): Same as above, but fork's skill files may differ. Same residual risk as `issue_comment` fork case — agent is sandboxed, pre-flight catches missing `SKILL.md`.
100+
- **`issue_comment`** (same-repo): Platform re-checks out PR branch — files already match, effectively a no-op
101+
- **`issue_comment`** (fork): Platform re-checks out fork branch after us, overwriting restored files. Agent is sandboxed; pre-flight in the prompt catches missing `SKILL.md`
99102

100103
### Anti-Patterns
101104

@@ -197,7 +200,7 @@ Manual triggers (`workflow_dispatch`, `issue_comment`) should bypass the gate. N
197200

198201
| What | Behavior | Workaround |
199202
|------|----------|------------|
200-
| User steps always before platform steps | Cannot run user code after `checkout_pr_branch.cjs` | Use `workflow_dispatch` for fork PRs; see [gh-aw#18481](https://github.com/github/gh-aw/issues/18481) |
203+
| User steps always before platform steps | Cannot run user code after `checkout_pr_branch.cjs` | For `issue_comment` fork PRs, accept sandboxed residual risk; see [gh-aw#18481](https://github.com/github/gh-aw/issues/18481) |
201204
| `--allow-all-tools` in lock.yml | Emitted by `gh aw compile` | Cannot override from `.md` source |
202205
| MCP integrity filtering | Fork PRs blocked as "unapproved" | Use `steps:` checkout instead of MCP |
203206
| `gh` CLI inside agent | Credentials scrubbed | Use `steps:` for API calls, or MCP tools |
@@ -215,8 +218,8 @@ Manual triggers (`workflow_dispatch`, `issue_comment`) should bypass the gate. N
215218
| Symptom | Cause | Fix |
216219
|---------|-------|-----|
217220
| Agent evaluates wrong PR | `workflow_dispatch` checks out workflow branch | Add `gh pr checkout` in `steps:` |
218-
| Agent can't find SKILL.md | Fork PR branch doesn't have `.github/skills/` | Agent posts "rebase or use `workflow_dispatch`" message; or rebase fork on `main` |
219-
| Fork PR rejected on `/evaluate-tests` | Fail-closed fork guard in `Checkout-GhAwPr.ps1` | Use `workflow_dispatch` with `pr_number` input instead |
221+
| Agent can't find SKILL.md | Fork PR branch doesn't include `.github/skills/` | Rebase fork on `main`, or use `workflow_dispatch` with `pr_number` input |
222+
| Fork PR skipped on `pull_request` | `forks: ["*"]` not in workflow frontmatter | Add `forks: ["*"]` under `pull_request:` in the `.md` source and recompile |
220223
| `gh` commands fail in agent | Credentials scrubbed inside container | Move to `steps:` section |
221224
| Lock file out of date | Forgot to recompile | Run `gh aw compile` |
222225
| Integrity filtering warning | MCP reading fork PR data | Expected, non-blocking |

.github/scripts/Checkout-GhAwPr.ps1

Lines changed: 10 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,13 @@
44
55
.DESCRIPTION
66
Checks out a PR branch and restores trusted agent infrastructure (skills,
7-
instructions) from the base branch. For issue_comment triggers, fork PRs
8-
are rejected (fail-closed) because the platform's checkout_pr_branch.cjs
9-
overwrites restored files after user steps. Fork PRs should use
10-
workflow_dispatch instead.
7+
instructions) from the base branch. Works for both same-repo and fork PRs.
8+
9+
This script is only invoked for workflow_dispatch triggers. For pull_request
10+
and issue_comment, the gh-aw platform's checkout_pr_branch.cjs handles PR
11+
checkout automatically (it runs as a platform step after all user steps).
12+
workflow_dispatch skips the platform checkout entirely, so this script is
13+
the only thing that gets the PR code onto disk.
1114
1215
SECURITY NOTE: This script checks out PR code onto disk. This is safe
1316
because NO subsequent user steps execute workspace code — the gh-aw
@@ -26,7 +29,6 @@
2629
PR_NUMBER - PR number to check out
2730
GITHUB_REPOSITORY - owner/repo (set by GitHub Actions)
2831
GITHUB_ENV - path to env file (set by GitHub Actions)
29-
GITHUB_EVENT_NAME - trigger type (set by GitHub Actions)
3032
#>
3133

3234
$ErrorActionPreference = 'Stop'
@@ -40,25 +42,6 @@ if (-not $env:PR_NUMBER -or $env:PR_NUMBER -eq '0') {
4042

4143
$PrNumber = $env:PR_NUMBER
4244

43-
# ── Fork guard (issue_comment only) ─────────────────────────────────────────
44-
# For issue_comment triggers, platform's checkout_pr_branch.cjs runs AFTER user
45-
# steps and re-checks out the fork branch, overwriting any restored skill/instruction
46-
# files. A fork could include a crafted SKILL.md that alters agent behavior.
47-
# Fail closed: if we can't verify origin, exit 1 (not 0).
48-
# Fork PRs can still be evaluated via workflow_dispatch (where platform checkout is skipped).
49-
50-
if ($env:GITHUB_EVENT_NAME -eq 'issue_comment') {
51-
$isFork = gh pr view $PrNumber --repo $env:GITHUB_REPOSITORY --json isCrossRepository --jq '.isCrossRepository' 2>&1
52-
if ($LASTEXITCODE -ne 0) {
53-
Write-Host "❌ Could not verify PR origin — failing closed"
54-
exit 1
55-
}
56-
if ($isFork -eq 'true') {
57-
Write-Host "::notice::Fork PR detected — /evaluate-tests via issue_comment is not supported for fork PRs. Use workflow_dispatch with pr_number=$PrNumber instead."
58-
exit 1
59-
}
60-
}
61-
6245
# ── Save base branch SHA ─────────────────────────────────────────────────────
6346
# Must be captured BEFORE checkout replaces HEAD.
6447
# Exported for potential use by downstream platform steps (e.g., checkout_pr_branch.cjs)
@@ -82,11 +65,9 @@ Write-Host "✅ Checked out PR #$PrNumber"
8265
git log --oneline -1
8366

8467
# ── Restore agent infrastructure from base branch ────────────────────────────
85-
# Best-effort restore of skill/instruction files from the base branch.
86-
# - workflow_dispatch: platform checkout is skipped, so this IS the final state
87-
# - issue_comment (same-repo): platform's checkout_pr_branch.cjs runs after and
88-
# overwrites, but files already match (same repo). Fork PRs are blocked above.
89-
# - pull_request (same-repo): files already exist, this is a no-op
68+
# This script only runs for workflow_dispatch (other triggers use the platform's
69+
# checkout_pr_branch.cjs instead). For workflow_dispatch the platform checkout is
70+
# skipped, so this restore IS the final workspace state.
9071
# rm -rf first to prevent fork-added files from surviving the restore.
9172

9273
if (Test-Path '.github/skills/') { Remove-Item -Recurse -Force '.github/skills/' }

.github/workflows/copilot-evaluate-tests.lock.yml

Lines changed: 9 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/copilot-evaluate-tests.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ description: Evaluates test quality, coverage, and appropriateness on PRs that a
33
on:
44
pull_request:
55
types: [opened, synchronize, reopened, ready_for_review]
6+
forks: ["*"]
67
paths:
78
- 'src/**/tests/**'
89
- 'src/**/test/**'
@@ -72,10 +73,14 @@ steps:
7273
echo "✅ Found test files to evaluate:"
7374
echo "$TEST_FILES" | head -20
7475
76+
# Only needed for workflow_dispatch — for pull_request and issue_comment,
77+
# the gh-aw platform's checkout_pr_branch.cjs handles PR checkout automatically.
78+
# workflow_dispatch skips the platform checkout entirely, so we must do it here.
7579
- name: Checkout PR and restore agent infrastructure
80+
if: github.event_name == 'workflow_dispatch'
7681
env:
7782
GH_TOKEN: ${{ github.token }}
78-
PR_NUMBER: ${{ github.event.pull_request.number || github.event.issue.number || inputs.pr_number }}
83+
PR_NUMBER: ${{ inputs.pr_number }}
7984
run: pwsh .github/scripts/Checkout-GhAwPr.ps1
8085
---
8186

0 commit comments

Comments
 (0)