Skip to content

Commit 890f70d

Browse files
committed
refactor: restructure CI workflow to optimize SauceLabs usage and build times
Completely restructure the validation workflow to separate concerns and respect SauceLabs concurrent session limits while maximizing parallelism. ## New Job Structure 1. **format-check** - Code style validation (unchanged) 2. **build** - Build entire project once, generate javadoc, cache artifacts 3. **unit-tests** - Run all unit tests using cached build (no SauceLabs) 4. **integration-tests** - Run only SauceLabs browser tests using cached build - Matrix: JUnit 4, 5, 6 variants - Sequential execution within workflow (max-parallel: 1) - Global concurrency control across all PRs ## Key Improvements **Build once, use everywhere:** - Build job runs `mvn clean install -DskipTests` once - Javadoc generated once in build job (not 3x in integration tests) - Artifacts cached with run-specific key: `github.run_id` - Both unit-tests and integration-tests restore from cache - Eliminates redundant builds (was building 3x, once per JUnit version) **Independent unit test execution:** - Runs `mvn test` separately from integration tests - No SauceLabs dependency (fast, cheap, can parallelize) - Separate test reports: surefire-reports (unit tests) - Added TestBench license setup for Pro feature validation **Optimized integration tests:** - Runs `mvn verify -Dsurefire.skip=true -pl <module> -am -P validation` - `-Dsurefire.skip=true` skips only unit tests (already ran in separate job) - Runs only failsafe integration tests (*IT.java files) - Uses full SauceLabs parallelism: `testsInParallel=5`, `forkCount=5` - Separate test reports: failsafe-reports (integration tests) - Removed javadoc generation (moved to build job) **SauceLabs concurrency control:** - Job-level concurrency group: `saucelabs-testbench` - Ensures only one set of integration tests runs globally - Matrix strategy runs JUnit 4/5/6 sequentially (max-parallel: 1) - Total concurrent sessions: exactly 5 (within SauceLabs limit) ## Resource Utilization **Multiple PRs can run in parallel:** - format-check: ✅ parallel across all PRs - build: ✅ parallel across all PRs (includes javadoc) - unit-tests: ✅ parallel across all PRs - integration-tests: ❌ queued globally (SauceLabs constraint only) **Performance benefits:** - Faster feedback: format/build/unit-tests run immediately in parallel - Faster builds: build once vs 3x (33% of previous time) - Javadoc generated once vs 3x (67% reduction) - Better SauceLabs utilization: full 5-session parallelism per test run - No wasted concurrent sessions from multiple workflows - Integration tests focus purely on browser testing (no javadoc overhead) ## Maven Cache Strategy Cache key includes `github.run_id` for per-workflow isolation: - build job: creates cache with `actions/cache@v4` - unit-tests job: restores cache with `actions/cache/restore@v4` - integration-tests job: restores cache with `actions/cache/restore@v4`
1 parent d23e0dc commit 890f70d

File tree

1 file changed

+120
-22
lines changed

1 file changed

+120
-22
lines changed

.github/workflows/validation.yml

Lines changed: 120 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -38,17 +38,15 @@ jobs:
3838
- name: Run formatter validation
3939
run: mvn formatter:validate --batch-mode --no-transfer-progress
4040

41-
build-and-test:
42-
name: Build and Test
41+
build:
42+
name: Build Project
4343
needs: format-check
4444
runs-on: ubuntu-latest
45-
timeout-minutes: 45
45+
timeout-minutes: 15
4646

4747
steps:
4848
- name: Checkout repository
4949
uses: actions/checkout@v4
50-
with:
51-
fetch-depth: 0
5250

5351
- name: Set up JDK ${{ env.JAVA_VERSION }}
5452
uses: actions/setup-java@v4
@@ -65,7 +63,7 @@ jobs:
6563
uses: actions/cache@v4
6664
with:
6765
path: ~/.m2/repository
68-
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
66+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}-${{ github.run_id }}
6967
restore-keys: |
7068
${{ runner.os }}-maven-
7169
@@ -75,6 +73,112 @@ jobs:
7573
echo "---> Clean maven cache"
7674
rm -rf ~/.m2/repository ~/.npm* ~/.pnpm*
7775
76+
- name: Build with Maven and Generate Javadoc
77+
run: |
78+
mvn clean install -DskipTests -Dtestbench.javadocs -B
79+
mvn javadoc:javadoc -B
80+
81+
unit-tests:
82+
name: Unit Tests
83+
needs: build
84+
runs-on: ubuntu-latest
85+
timeout-minutes: 15
86+
87+
steps:
88+
- name: Checkout repository
89+
uses: actions/checkout@v4
90+
91+
- name: Set up JDK ${{ env.JAVA_VERSION }}
92+
uses: actions/setup-java@v4
93+
with:
94+
java-version: ${{ env.JAVA_VERSION }}
95+
distribution: "temurin"
96+
97+
- name: Restore Maven cache
98+
uses: actions/cache/restore@v4
99+
with:
100+
path: ~/.m2/repository
101+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}-${{ github.run_id }}
102+
103+
- name: Set TB License
104+
run: |
105+
TB_LICENSE=${{secrets.TB_LICENSE}}
106+
mkdir -p ~/.vaadin/
107+
echo '{"username":"'`echo $TB_LICENSE | cut -d / -f1`'","proKey":"'`echo $TB_LICENSE | cut -d / -f2`'"}' > ~/.vaadin/proKey
108+
109+
- name: Run unit tests
110+
run: |
111+
mvn test -B
112+
113+
- name: Upload test reports on failure
114+
if: failure()
115+
uses: actions/upload-artifact@v4
116+
with:
117+
name: unit-test-reports-${{ github.run_id }}
118+
path: |
119+
**/target/surefire-reports/
120+
retention-days: 7
121+
122+
- name: Upload test results
123+
if: always()
124+
uses: actions/upload-artifact@v4
125+
with:
126+
name: unit-test-results-${{ github.run_id }}
127+
path: |
128+
**/target/surefire-reports/TEST-*.xml
129+
retention-days: 7
130+
131+
- name: Publish unit test results
132+
if: always()
133+
uses: EnricoMi/publish-unit-test-result-action@v2
134+
with:
135+
files: |
136+
**/target/surefire-reports/TEST-*.xml
137+
check_name: Unit Test Results
138+
comment_mode: failures
139+
140+
integration-tests:
141+
name: Integration Tests (${{ matrix.name }})
142+
needs: build
143+
runs-on: ubuntu-latest
144+
timeout-minutes: 45
145+
concurrency:
146+
group: saucelabs-testbench # Global queue for SauceLabs tests only
147+
cancel-in-progress: false
148+
strategy:
149+
max-parallel: 1 # Only one JUnit version at a time to stay within SauceLabs limit
150+
matrix:
151+
include:
152+
- name: JUnit 4
153+
module: vaadin-testbench-integration-tests
154+
- name: JUnit 5
155+
module: vaadin-testbench-integration-tests-junit5
156+
- name: JUnit 6
157+
module: vaadin-testbench-integration-tests-junit6
158+
159+
steps:
160+
- name: Checkout repository
161+
uses: actions/checkout@v4
162+
with:
163+
fetch-depth: 0
164+
165+
- name: Set up JDK ${{ env.JAVA_VERSION }}
166+
uses: actions/setup-java@v4
167+
with:
168+
java-version: ${{ env.JAVA_VERSION }}
169+
distribution: "temurin"
170+
171+
- name: Set up Node.js
172+
uses: actions/setup-node@v4
173+
with:
174+
node-version: "20"
175+
176+
- name: Restore Maven cache
177+
uses: actions/cache/restore@v4
178+
with:
179+
path: ~/.m2/repository
180+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}-${{ github.run_id }}
181+
78182
- name: Set up Sauce Labs tunnel
79183
uses: saucelabs/[email protected]
80184
with:
@@ -91,19 +195,16 @@ jobs:
91195
mkdir -p ~/.vaadin/
92196
echo '{"username":"'`echo $TB_LICENSE | cut -d / -f1`'","proKey":"'`echo $TB_LICENSE | cut -d / -f2`'"}' > ~/.vaadin/proKey
93197
94-
- name: Build with Maven
95-
run: |
96-
mvn clean install -DskipTests -B
97-
98-
- name: Run Tests and Generate Javadoc
198+
- name: Run Integration Tests
99199
env:
100200
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
101201
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
102202
SAUCE_TUNNEL_ID: ${{ github.run_id }}-${{ github.run_number }}
103203
run: |
104-
mvn javadoc:javadoc verify \
204+
mvn verify \
205+
-pl ${{ matrix.module }} -am \
105206
-P validation \
106-
-Dtestbench.javadocs \
207+
-Dsurefire.skip=true \
107208
-Dsystem.com.vaadin.testbench.Parameters.testsInParallel=5 \
108209
-Dsystem.com.vaadin.testbench.Parameters.maxAttempts=2 \
109210
-Dcom.vaadin.testbench.Parameters.hubHostname=localhost \
@@ -117,37 +218,34 @@ jobs:
117218
if: failure()
118219
uses: actions/upload-artifact@v4
119220
with:
120-
name: error-screenshots-${{ github.run_id }}
221+
name: error-screenshots-${{ matrix.name }}-${{ github.run_id }}
121222
path: |
122223
**/error-screenshots/**
123224
retention-days: 7
124-
225+
125226
- name: Upload test reports on failure
126227
if: failure()
127228
uses: actions/upload-artifact@v4
128229
with:
129-
name: test-reports-${{ github.run_id }}
230+
name: integration-test-reports-${{ matrix.name }}-${{ github.run_id }}
130231
path: |
131-
**/target/surefire-reports/
132232
**/target/failsafe-reports/
133233
retention-days: 7
134234

135235
- name: Upload test results
136236
if: always()
137237
uses: actions/upload-artifact@v4
138238
with:
139-
name: test-results-${{ github.run_id }}
239+
name: integration-test-results-${{ matrix.name }}-${{ github.run_id }}
140240
path: |
141-
**/target/surefire-reports/TEST-*.xml
142241
**/target/failsafe-reports/TEST-*.xml
143242
retention-days: 7
144243

145-
- name: Publish test results
244+
- name: Publish integration test results
146245
if: always()
147246
uses: EnricoMi/publish-unit-test-result-action@v2
148247
with:
149248
files: |
150-
**/target/surefire-reports/TEST-*.xml
151249
**/target/failsafe-reports/TEST-*.xml
152-
check_name: Test Results
250+
check_name: Integration Test Results (${{ matrix.name }})
153251
comment_mode: failures

0 commit comments

Comments
 (0)