Skip to content

Commit dfe50a5

Browse files
Fix skill eval change detection to compare full PR diff against base
The previous logic used 'git diff-tree -r HEAD' which only compares the latest commit against its parent. On PRs with multiple commits, this misses skill changes from earlier commits and incorrectly skips evals. Now uses the PR base SHA (github.event.pull_request.base.sha) with three-dot diff to detect ALL skill file changes across the entire PR. Also uses fetch-depth: 0 so the base SHA is available. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 67da918 commit dfe50a5

1 file changed

Lines changed: 11 additions & 2 deletions

File tree

.github/workflows/skill-eval.yml

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,14 +61,23 @@ jobs:
6161
steps:
6262
- uses: actions/checkout@v4
6363
with:
64-
fetch-depth: 2
64+
fetch-depth: 0
6565

6666
- name: Check if skill files changed
6767
id: changes
6868
run: |
6969
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
7070
echo "SHOULD_RUN=true" >> $GITHUB_OUTPUT
71-
elif git diff-tree --no-commit-id --name-only -r HEAD | grep -q "^\.github/skills/"; then
71+
elif [ "${{ github.event_name }}" = "pull_request" ]; then
72+
# Compare PR head against base branch to detect all changes in the PR
73+
BASE_SHA="${{ github.event.pull_request.base.sha }}"
74+
if git diff --name-only "$BASE_SHA"...HEAD | grep -q "^\.github/skills/"; then
75+
echo "SHOULD_RUN=true" >> $GITHUB_OUTPUT
76+
else
77+
echo "SHOULD_RUN=false" >> $GITHUB_OUTPUT
78+
echo "No skill file changes detected in PR — skipping evals"
79+
fi
80+
elif git diff --name-only HEAD~1 HEAD | grep -q "^\.github/skills/"; then
7281
echo "SHOULD_RUN=true" >> $GITHUB_OUTPUT
7382
else
7483
echo "SHOULD_RUN=false" >> $GITHUB_OUTPUT

0 commit comments

Comments
 (0)