Skip to content

Commit 3a749a8

Browse files
batonogovbatonogov
authored
test(lsp): schedule real SourceKit-LSP smoke (#1453)
1 parent 6a36104 commit 3a749a8

6 files changed

Lines changed: 495 additions & 32 deletions

File tree

‎.github/SOURCEKIT_LSP_SMOKE.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# SourceKit-LSP smoke testing
2+
3+
Pine runs its real SourceKit-LSP integration suite every day at 06:00 UTC in
4+
`.github/workflows/nightly-sourcekit-lsp.yml`. The workflow can also be started
5+
manually. It is scheduled-only while runner stability is measured, so it does
6+
not block pull requests or releases yet; a failed smoke run still fails its
7+
job.
8+
9+
The job resolves `sourcekit-lsp` through `xcrun` after selecting Xcode and
10+
fails before testing if the executable is missing or belongs to another
11+
developer directory. It runs only
12+
`PineTests/SourceKitLSPIntegrationTests`, without retry-on-failure. The suite
13+
bounds initialize, diagnostics, document synchronization, hover, definition,
14+
graceful shutdown, and forced cleanup. It also verifies that failure,
15+
cancellation, and timeout paths reap the real language-server process.
16+
17+
Every fixture creates an isolated temporary Swift package. The server receives
18+
an allowlisted environment with private HOME, CFFIXED_USER_HOME, XDG config and
19+
cache, SwiftPM config and module cache, Clang module cache, scratch, and TMPDIR
20+
paths. It does not inherit the runner user's configuration. Fixture project and
21+
cache directories are removed after each test. Captured server stderr is
22+
written to `SourceKitLSPArtifacts` so it survives fixture cleanup.
23+
24+
The workflow has a 30-minute job bound, enables per-test timeouts, validates
25+
that the fresh `.xcresult` contains executed passing tests, and always uploads
26+
the result bundle, `xcodebuild` log, and server stderr for 14 days.
27+
28+
## Promotion threshold
29+
30+
Make this lane a required main and release check only after 30 consecutive scheduled runs meet all of these conditions:
31+
32+
- zero functional failures, timeouts, or leaked SourceKit-LSP processes;
33+
- at least 95% infrastructure completion, excluding confirmed runner outages;
34+
- no test skip caused by a missing or mismatched Xcode toolchain component.
35+
36+
Reset the 30-run window after a smoke-test expansion or a failure. Review the
37+
threshold before promoting the lane.
38+
39+
## Local run
40+
41+
Select Xcode, resolve `xcrun --find sourcekit-lsp`, and run:
42+
43+
```sh
44+
PINE_RUN_SOURCEKIT_LSP_SMOKE=1 \
45+
PINE_SOURCEKIT_LSP_EXECUTABLE=/absolute/path/to/sourcekit-lsp \
46+
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
47+
xcodebuild test \
48+
-project Pine.xcodeproj \
49+
-scheme Pine \
50+
-destination 'platform=macOS' \
51+
-only-testing:PineTests/SourceKitLSPIntegrationTests \
52+
-parallel-testing-enabled NO \
53+
-test-timeouts-enabled YES
54+
```

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,9 @@ jobs:
8585
- name: Tests for sanitizer workflow
8686
run: bash scripts/tests/test-sanitizer-workflow.sh
8787

88+
- name: Tests for SourceKit-LSP smoke workflow
89+
run: bash scripts/tests/test-sourcekit-lsp-workflow.sh
90+
8891
- name: Tests for DMG packaging scripts
8992
run: bash scripts/tests/test-dmg-packaging.sh
9093

‎.github/workflows/nightly-sourcekit-lsp.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# NOTE: All third-party actions are pinned by full commit SHA for supply-chain safety.
2+
# To update an action: find the new version's commit SHA on GitHub (Tags -> verify commit),
3+
# replace the SHA, and keep the "# vX" comment in sync.
4+
name: Nightly SourceKit-LSP Smoke
5+
6+
on:
7+
schedule:
8+
# Run after the other nightly test lanes to reduce runner contention.
9+
- cron: '0 6 * * *'
10+
workflow_dispatch:
11+
12+
permissions:
13+
contents: read
14+
15+
concurrency:
16+
group: nightly-sourcekit-lsp
17+
cancel-in-progress: false
18+
19+
# This scheduled/manual lane intentionally starts outside blocking PR CI.
20+
# Promotion criteria live in .github/SOURCEKIT_LSP_SMOKE.md.
21+
jobs:
22+
sourcekit-lsp-smoke:
23+
name: Real SourceKit-LSP Smoke
24+
runs-on: macos-26
25+
timeout-minutes: 30
26+
steps:
27+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
28+
29+
- name: Select Xcode
30+
uses: ./.github/actions/select-xcode
31+
32+
- name: Ensure Metal Toolchain
33+
run: bash scripts/ensure-metal-toolchain.sh
34+
35+
- name: Locate SourceKit-LSP in Selected Xcode
36+
shell: bash
37+
run: |
38+
set -euo pipefail
39+
sourcekit_lsp="$(xcrun --find sourcekit-lsp 2>/dev/null || true)"
40+
if [ -z "$sourcekit_lsp" ] || [ ! -x "$sourcekit_lsp" ]; then
41+
echo "::error::The selected Xcode toolchain does not provide an executable sourcekit-lsp"
42+
exit 1
43+
fi
44+
case "$sourcekit_lsp" in
45+
"$DEVELOPER_DIR"/*) ;;
46+
*)
47+
echo "::error::sourcekit-lsp resolved outside the selected Xcode: $sourcekit_lsp"
48+
exit 1
49+
;;
50+
esac
51+
echo "Using sourcekit-lsp at $sourcekit_lsp"
52+
echo "PINE_SOURCEKIT_LSP_EXECUTABLE=$sourcekit_lsp" >> "$GITHUB_ENV"
53+
54+
- name: Cache SPM Dependencies
55+
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
56+
with:
57+
path: DerivedData/SourcePackages
58+
key: spm-${{ runner.os }}-${{ hashFiles('Pine.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved') }}
59+
restore-keys: |
60+
spm-${{ runner.os }}-
61+
62+
- name: Run Real SourceKit-LSP Smoke
63+
shell: bash
64+
env:
65+
PINE_RUN_SOURCEKIT_LSP_SMOKE: "1"
66+
PINE_SOURCEKIT_LSP_ARTIFACTS_DIR: ${{ github.workspace }}/SourceKitLSPArtifacts
67+
run: |
68+
set -euo pipefail
69+
mkdir -p SourceKitLSPArtifacts
70+
result_bundle="SourceKitLSPResults.xcresult"
71+
console_log="SourceKitLSPArtifacts/xcodebuild.log"
72+
test_started_at="$(date +%s)"
73+
set +e
74+
xcodebuild test \
75+
-project Pine.xcodeproj \
76+
-scheme Pine \
77+
-destination "platform=macOS" \
78+
-derivedDataPath DerivedData \
79+
-only-testing:PineTests/SourceKitLSPIntegrationTests \
80+
-enableCodeCoverage NO \
81+
-parallel-testing-enabled NO \
82+
-test-timeouts-enabled YES \
83+
-resultBundlePath "$result_bundle" \
84+
CODE_SIGN_IDENTITY=- \
85+
CODE_SIGNING_ALLOWED=NO \
86+
CODE_SIGNING_REQUIRED=NO \
87+
PINE_RUN_SOURCEKIT_LSP_SMOKE=1 \
88+
PINE_SOURCEKIT_LSP_EXECUTABLE="$PINE_SOURCEKIT_LSP_EXECUTABLE" \
89+
PINE_SOURCEKIT_LSP_ARTIFACTS_DIR="$PINE_SOURCEKIT_LSP_ARTIFACTS_DIR" \
90+
2>&1 | tee "$console_log"
91+
test_exit=${PIPESTATUS[0]}
92+
set -e
93+
94+
python3 .github/scripts/validate_test_results.py \
95+
"$result_bundle" \
96+
--xcodebuild-exit "$test_exit" \
97+
--started-at "$test_started_at" \
98+
--github-summary
99+
100+
- name: Upload SourceKit-LSP Results
101+
if: always()
102+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
103+
with:
104+
name: SourceKitLSP-${{ github.run_id }}-${{ github.run_attempt }}
105+
path: |
106+
SourceKitLSPResults.xcresult
107+
SourceKitLSPArtifacts
108+
if-no-files-found: warn
109+
retention-days: 14

‎Pine.xcodeproj/xcshareddata/xcschemes/Pine.xcscheme‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,16 @@
4949
value = "$(PINE_RUN_SOURCEKIT_LSP_SMOKE)"
5050
isEnabled = "YES">
5151
</EnvironmentVariable>
52+
<EnvironmentVariable
53+
key = "PINE_SOURCEKIT_LSP_EXECUTABLE"
54+
value = "$(PINE_SOURCEKIT_LSP_EXECUTABLE)"
55+
isEnabled = "YES">
56+
</EnvironmentVariable>
57+
<EnvironmentVariable
58+
key = "PINE_SOURCEKIT_LSP_ARTIFACTS_DIR"
59+
value = "$(PINE_SOURCEKIT_LSP_ARTIFACTS_DIR)"
60+
isEnabled = "YES">
61+
</EnvironmentVariable>
5262
<EnvironmentVariable
5363
key = "PINE_RECORD_SNAPSHOTS"
5464
value = "$(PINE_RECORD_SNAPSHOTS)"

0 commit comments

Comments
 (0)