Skip to content

Commit b78e114

Browse files
lennessyyclaude
lennessyy
and
claude
authored
feat: visual comparison CI workflow (#3884)
* feat: first stab at visual comparison CI workflow * add visual test files * add visual comparison workflow * fix visual comparison * longer timeout * fix: add explicit permissions to all workflow jobs Add explicit permissions blocks to all jobs in the visual comparison workflows to address Semgrep security findings. Following least-privilege principle: - visual-comparison.yml: Added permissions for contents, actions, and pull-requests - screenshot-capture.yml: Added permissions for contents and actions - cleanup-report.yml: Added permissions for contents write 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: prevent script injection in GitHub context usage Move all GitHub context variables in run steps to environment variables to prevent script injection attacks. This addresses Semgrep findings for: - visual-comparison.yml: github.base_ref, github.head_ref, github.ref_name, github.run_id, github.run_attempt - cleanup-report.yml: github.event.ref Branch names and other GitHub context data can contain special characters that could lead to command injection when directly interpolated in bash. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: use visual-comparison label and remove branch ignores - Change trigger label from 'visual-tests' to 'visual-comparison' - Remove branches-ignore for version-* and docs-rel-* (not applicable to this repo) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: remove duplicate @playwright/test from dependencies Remove @playwright/test from dependencies to resolve conflict with devDependencies. Playwright is a testing tool and should only be in devDependencies (^1.55.1). This fixes the "Your lockfile needs to be updated" error in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: create gh-pages branch if it doesn't exist Add step to automatically create the gh-pages branch when it doesn't exist, allowing the workflow to publish HTML reports on first run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: improve gh-pages branch creation logic Check if gh-pages branch exists before attempting checkout or creation. This avoids the git state issues that were causing the previous approach to fail. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: set git user before creating gh-pages branch Move git config step before the commit to avoid "empty ident name" error. Also set git user for both the creation and existing branch paths. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: explicitly specify remote and branch in git commands - Use `git push -u origin gh-pages` when creating the branch - Use explicit `origin gh-pages` in pull/push commands to avoid tracking issues This fixes the "no tracking information for the current branch" error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: handle race condition in gh-pages branch creation - Make branch creation non-failing if branch already exists (race condition) - Always checkout gh-pages after creation attempt, simplifying the flow - Set git user once for all cases The gh-pages branch now exists, so future runs will skip creation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Replace GitHub Pages with Vercel for visual test reports Simplifies the deployment process by using Vercel instead of GitHub Pages: - Removes gh-pages branch management and race condition handling - Eliminates 120-second DNS propagation wait - Removes git operations and conflict resolution logic - Simplifies permissions (no longer needs contents: write) - Provides instant deployment URLs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add bright red navbar for visual comparison testing Makes the navbar bright red (#ff0000) with white text to create an obvious visual difference for testing the Vercel deployment workflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Revert red navbar test changes Removing test CSS as baselines need to be established first before visual comparison testing can work properly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * docs: add to utilities.md --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent ffcfe6f commit b78e114

15 files changed

Lines changed: 828 additions & 49 deletions

‎.github/workflows/cleanup-report.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Delete Visual Tests Reports
2+
3+
on:
4+
delete:
5+
branches-ignore: [master, main, gh-pages]
6+
7+
concurrency:
8+
group: ${{ github.event.ref }}
9+
cancel-in-progress: true
10+
11+
jobs:
12+
delete_reports:
13+
name: Delete Reports
14+
runs-on: ubuntu-latest
15+
permissions:
16+
contents: write
17+
env:
18+
BRANCH_REPORTS_DIR: reports/${{ github.event.ref }}
19+
steps:
20+
- name: Checkout GitHub Pages Branch
21+
uses: actions/checkout@v5
22+
with:
23+
ref: gh-pages
24+
25+
- name: Set Git User
26+
run: |
27+
git config --global user.name "github-actions[bot]"
28+
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
29+
30+
- name: Check for workflow reports
31+
run: |
32+
if [ -d "$BRANCH_REPORTS_DIR" ] && [ "$(ls -A $BRANCH_REPORTS_DIR)" ]; then
33+
echo "BRANCH_REPORTS_EXIST=true" >> "$GITHUB_ENV"
34+
else
35+
echo "BRANCH_REPORTS_EXIST=false" >> "$GITHUB_ENV"
36+
fi
37+
38+
- name: Delete reports from repo for branch
39+
if: ${{ env.BRANCH_REPORTS_EXIST == 'true' }}
40+
timeout-minutes: 3
41+
env:
42+
DELETED_BRANCH: ${{ github.event.ref }}
43+
run: |
44+
cd "$(dirname "$BRANCH_REPORTS_DIR")"
45+
rm -rf "${DELETED_BRANCH}"
46+
git add .
47+
git commit -m "workflow: remove all reports for branch ${DELETED_BRANCH}"
48+
while true; do
49+
git pull --rebase
50+
if [ $? -ne 0 ]; then
51+
echo "Failed to rebase. Please review manually."
52+
exit 1
53+
fi
54+
git push
55+
if [ $? -eq 0 ]; then
56+
echo "Successfully pushed HTML reports to repo."
57+
exit 0
58+
fi
59+
done

‎.github/workflows/screenshot-capture.yml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
name: Screenshot Capture
2+
3+
on:
4+
schedule:
5+
- cron: '0 0 * * 0' # Weekly on Sunday at midnight UTC
6+
workflow_dispatch:
7+
8+
env:
9+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
10+
PALETTE_API_KEY: ${{ secrets.PALETTE_API_KEY }}
11+
FULLSTORY_ORGID: ${{ secrets.FULLSTORY_ORGID }}
12+
ALGOLIA_ADMIN_KEY: ${{ secrets.ALGOLIA_ADMIN_KEY }}
13+
ALGOLIA_APP_ID: ${{ secrets.ALGOLIA_APP_ID }}
14+
ALGOLIA_SEARCH_KEY: ${{ secrets.ALGOLIA_SEARCH_KEY }}
15+
ALGOLIA_INDEX_NAME: ${{ secrets.ALGOLIA_INDEX_NAME }}
16+
DISABLE_PACKS_INTEGRATIONS: ${{ secrets.DISABLE_PACKS_INTEGRATIONS }}
17+
DISABLE_SECURITY_INTEGRATIONS: ${{ secrets.DISABLE_SECURITY_INTEGRATIONS }}
18+
SHOW_LAST_UPDATE_TIME: ${{ secrets.SHOW_LAST_UPDATE_TIME }}
19+
DSO_AUTH_TOKEN: ${{ secrets.DSO_AUTH_TOKEN }}
20+
SEERS_CMP_KEY: ${{ secrets.SEERS_CMP_KEY }}
21+
22+
jobs:
23+
create-assets:
24+
name: asset-builds
25+
runs-on: ubuntu-latest
26+
permissions:
27+
contents: read
28+
steps:
29+
- name: Checkout Repository
30+
uses: actions/checkout@v5
31+
32+
- name: Setup Node.js Environment
33+
uses: actions/setup-node@v4
34+
with:
35+
node-version: "20"
36+
cache: 'yarn'
37+
38+
- name: Install Dependencies
39+
run: yarn install --frozen-lockfile
40+
41+
- name: Install Playwright browsers
42+
run: npx playwright install --with-deps chromium
43+
44+
- name: Build Website
45+
run: |
46+
touch .env
47+
yarn build
48+
49+
- name: Upload Build
50+
uses: actions/upload-artifact@v4
51+
with:
52+
name: build
53+
path: build/
54+
if-no-files-found: error
55+
retention-days: 1
56+
57+
visual-snapshots:
58+
runs-on: ubuntu-latest
59+
needs: [create-assets]
60+
permissions:
61+
contents: read
62+
actions: read
63+
strategy:
64+
fail-fast: false
65+
matrix:
66+
shardIndex: [1, 2, 3, 4]
67+
shardTotal: [4]
68+
steps:
69+
- name: Checkout Repository
70+
uses: actions/checkout@v5
71+
72+
- name: Setup Node.js Environment
73+
uses: actions/setup-node@v4
74+
with:
75+
node-version: "20"
76+
cache: 'yarn'
77+
78+
- name: Install Dependencies
79+
uses: Wandalen/wretry.action@v3
80+
with:
81+
command: yarn install --frozen-lockfile
82+
attempt_limit: 3
83+
attempt_delay: 60000
84+
85+
- name: Install Playwright browsers
86+
run: npx playwright install --with-deps chromium
87+
88+
- name: Download build artifact
89+
uses: Wandalen/wretry.action@v3
90+
with:
91+
attempt_limit: 3
92+
attempt_delay: 60000
93+
action: actions/download-artifact@v5
94+
with: |
95+
name: build
96+
path: build
97+
98+
- name: Take Screenshots with Playwright
99+
env:
100+
PLAYWRIGHT_SNAPSHOT_DIR: ${{ github.workspace }}/screenshots
101+
run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --update-snapshots || exit 0
102+
103+
- name: Upload Screenshots
104+
uses: actions/upload-artifact@v4
105+
with:
106+
name: screenshots-${{ matrix.shardIndex }}
107+
path: screenshots/
108+
if-no-files-found: error
109+
retention-days: 3
110+
111+
merge-snapshots:
112+
name: Merge Screenshots
113+
runs-on: ubuntu-latest
114+
needs: [visual-snapshots]
115+
permissions:
116+
actions: read
117+
steps:
118+
- name: Download screenshots
119+
uses: actions/download-artifact@v5
120+
with:
121+
path: screenshots
122+
pattern: screenshots-*
123+
merge-multiple: true
124+
125+
- name: Upload merged screenshots
126+
uses: actions/upload-artifact@v4
127+
with:
128+
name: screenshots
129+
path: screenshots/
130+
if-no-files-found: error
131+
retention-days: 14

0 commit comments

Comments
 (0)