-
-
Notifications
You must be signed in to change notification settings - Fork 116
test(integration): add integration tests with emulate.dev #314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 17 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
b5def0f
docs(track): add emulate-integration-test-20260326
amondnet 05982d3
chore(track): emulate-integration-test-20260326 start implementation
amondnet 99a08db
test(integration): add emulate.dev infrastructure with Vitest Projects
amondnet 764a0f2
test(integration): add seed config and shared test helpers
amondnet c03ba8b
test(integration): add Vercel deployment API contract tests
amondnet 3f27ec5
test(integration): add Vercel domain and alias API contract tests
amondnet 8f33a3a
test(integration): add GitHub comment integration tests
amondnet 074a8e2
ci: add integration test job to CI workflow
amondnet beb36b8
docs(track): emulate-integration-test-20260326 sync tech-stack.md
amondnet c938bc7
chore: enable vercel plugin in claude settings
amondnet 3e10052
test(integration): address review findings
amondnet 82a6c24
docs(track): add retrospective and PR reference
amondnet 2e7b21e
refactor: extract GitHubContext to decouple from global singleton
amondnet 840e778
test(integration): rewrite to test actual action functions
amondnet 889b3f2
refactor: extract VercelClient port with CLI and API adapters
amondnet 198f169
test(integration): add VercelApiClient integration tests
amondnet 08ae401
chore: apply AI code review suggestions
amondnet b9ae17e
chore: fix remaining plan.md inconsistencies with direct-fetch approach
amondnet 021a335
fix(integration): use fixed default ports 4000/4001 in emulator setup
Copilot 0fd42e4
fix: revert dynamic ports and fix alias skipIf timing
amondnet bd4d1ac
chore: apply AI code review suggestions
amondnet e2309f3
fix: restore commitlint 200-char limit and getPortFromEnv default ports
amondnet ff1e1b3
fix: remove scope from vercel.json for northstar account compatibility
amondnet File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 changes: 9 additions & 0 deletions
9
.please/docs/tracks/active/emulate-integration-test-20260326/metadata.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "track_id": "emulate-integration-test-20260326", | ||
| "type": "feature", | ||
| "status": "in_progress", | ||
| "created_at": "2026-03-26T17:20:28+09:00", | ||
| "updated_at": "2026-03-26T17:30:00+09:00", | ||
| "issue": "", | ||
| "pr": "#314" | ||
| } |
175 changes: 175 additions & 0 deletions
175
.please/docs/tracks/active/emulate-integration-test-20260326/plan.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| # Plan: Integration Tests with emulate.dev | ||
|
|
||
| > Track: emulate-integration-test-20260326 | ||
| > Spec: [spec.md](./spec.md) | ||
|
|
||
| ## Overview | ||
|
|
||
| - **Source**: /please:plan | ||
| - **Track**: emulate-integration-test-20260326 | ||
| - **Created**: 2026-03-26 | ||
| - **Approach**: Two-layer testing (direct fetch for Vercel API + GitHub Octokit) against emulate.dev | ||
|
|
||
| ## Purpose | ||
|
|
||
| Add integration tests using [emulate.dev](https://emulate.dev) — a local, stateful API emulator from Vercel Labs. This enables end-to-end validation of API interactions without hitting production endpoints or requiring network access. | ||
|
|
||
| ## Context | ||
|
|
||
| ### Current State | ||
| - Unit tests use Vitest with `vi.mock()` for all external dependencies | ||
| - No integration tests exist; example workflows serve as manual E2E validation | ||
| - All Vercel interactions go through CLI (`npx vercel`), not direct HTTP | ||
| - GitHub API uses Octokit via `@actions/github.getOctokit()` with hardcoded base URL | ||
|
|
||
| ### Key Constraints | ||
| - emulate.dev provides stateful Vercel API (port 4000) and GitHub API (port 4001) | ||
| - `@vercel/sdk` supports `serverURL` option for custom API endpoints | ||
| - `@actions/github.getOctokit()` supports `baseUrl` option for custom endpoints | ||
|
|
||
| ## Architecture Decision | ||
|
|
||
| **Two test layers against emulate.dev using official SDK clients:** | ||
|
|
||
| 1. **Vercel API integration tests** — Use direct `fetch` calls with `Authorization: Bearer <token>` headers against `http://localhost:4000` to test deployment creation, retrieval, and domain/alias management. (`@vercel/sdk` was evaluated but dropped due to strict Zod validation that rejects emulate.dev responses.) | ||
|
|
||
| 2. **GitHub API integration tests** — Create Octokit client with `baseUrl: http://localhost:4001` and test the action's comment functions (`createCommentOnPullRequest`, `createCommentOnCommit`) against emulate.dev's stateful GitHub API. | ||
|
|
||
| **Why direct fetch instead of `@vercel/sdk`?** The SDK's Zod validation layer is too strict for emulate.dev's responses — it rejects fields the emulator omits or approximates. Direct `fetch` with manual JSON parsing provides the same coverage without validation friction. | ||
|
|
||
| **Why Vitest Projects?** [Vitest Projects](https://vitest.dev/guide/projects) allow defining `unit` and `integration` as separate projects in a single `vitest.config.ts`, with independent `include` patterns, `globalSetup`, and timeouts. This is cleaner than maintaining separate config files. | ||
|
|
||
| ## Tasks | ||
|
|
||
| ### Phase 1: Infrastructure Setup | ||
|
|
||
| - [x] T-1: Install dependencies and create Vitest globalSetup | ||
| - `pnpm add -D emulate` (note: `@vercel/sdk` was evaluated but dropped — see Architecture Decision section) | ||
| - Create `src/__integration__/global-setup.ts` using emulate's programmatic API (`createEmulator`) | ||
| - Start Vercel (port 4000) and GitHub (port 4001) services in `setup()` | ||
| - Tear down in `teardown()` | ||
| - **Files**: `src/__integration__/global-setup.ts`, `package.json` | ||
| - **Verify**: `pnpm test:integration` starts emulate.dev and exits cleanly | ||
|
|
||
| - [x] T-2: Configure Vitest Projects for unit and integration test separation | ||
| - Refactor `vitest.config.ts` to use [Vitest Projects](https://vitest.dev/guide/projects) | ||
| - Define `unit` project: `include: ['src/__tests__/**/*.test.ts']`, existing coverage thresholds | ||
| - Define `integration` project: `include: ['src/__integration__/**/*.test.ts']`, `globalSetup: 'src/__integration__/global-setup.ts'`, extended timeouts | ||
| - Add `test:unit` and `test:integration` scripts using `vitest run --project unit` / `--project integration` | ||
| - Keep `test` script running all projects (both unit + integration) | ||
| - **Files**: `vitest.config.ts`, `package.json` | ||
| - **Verify**: `pnpm test:integration` runs only integration tests, `pnpm test:unit` runs only unit tests | ||
|
|
||
| - [x] T-3: Create emulate.config.yaml seed data | ||
| - Define tokens, Vercel users/teams/projects, GitHub users/repos | ||
| - Seed data should match action's typical usage (org ID, project ID, repo owner/name) | ||
| - Create shared test helpers (`src/__integration__/helpers.ts`) with Vercel SDK and Octokit client factories | ||
| - **Files**: `emulate.config.yaml`, `src/__integration__/helpers.ts` | ||
| - **Verify**: emulate.dev starts with seed data populated | ||
|
|
||
| ### Phase 2: Vercel API Integration Tests | ||
|
|
||
| - [x] T-4: Test deployment creation and retrieval via direct fetch | ||
| - Use direct `fetch` with `Authorization: Bearer <token>` against `http://localhost:4000` | ||
| - `POST /v13/deployments` — create deployment, verify response structure | ||
| - `GET /v13/deployments/:id` — retrieve by ID via `VercelApiClient.inspect()` | ||
| - **Files**: `src/__integration__/vercel-api.test.ts` | ||
| - **Verify**: Deployment lifecycle endpoints return expected shapes | ||
|
|
||
| - [x] T-5: Test domain and alias management via direct fetch | ||
| - `POST /v9/projects/:id/domains` — add domain to project | ||
| - `GET /v9/projects/:id/domains` — list project domains | ||
| - `POST /v9/projects/:id/domains/:domain/verify` — verify domain | ||
| - `POST /v2/deployments/:id/aliases` — assign alias (skips gracefully if emulator returns 404) | ||
| - **Files**: `src/__integration__/vercel-api.test.ts` | ||
| - **Verify**: Domain creation, listing, verification, and alias assignment work via direct fetch | ||
|
|
||
| ### Phase 3: GitHub API Integration Tests | ||
|
|
||
| - [x] T-6: Test PR comment creation and update | ||
| - Create Octokit client with `baseUrl: http://localhost:4001` | ||
| - Test `issues.createComment` → verify comment appears in `issues.listComments` | ||
| - Test `issues.updateComment` → verify comment body is updated | ||
| - Test the find-previous-comment pattern used by the action | ||
| - **Files**: `src/__integration__/github-pr-comments.test.ts` | ||
| - **Verify**: Full PR comment create/find/update cycle works against emulated API | ||
|
|
||
| - [x] T-7: Test commit comment creation and update | ||
| - Test `repos.createCommitComment` → verify in `repos.listCommentsForCommit` | ||
| - Test `repos.updateCommitComment` → verify body update | ||
| - Test the find-previous-comment pattern for push events | ||
| - **Files**: `src/__integration__/github-commit-comments.test.ts` | ||
| - **Verify**: Full commit comment create/find/update cycle works against emulated API | ||
|
|
||
| ### Phase 4: CI Integration | ||
|
|
||
| - [x] T-8: Add integration test job to CI workflow | ||
| - Add `test:integration` job to `.github/workflows/ci.yml` | ||
| - Same Node.js/pnpm setup as existing test job | ||
| - Runs `pnpm test:integration` | ||
| - **Files**: `.github/workflows/ci.yml` | ||
| - **Verify**: CI pipeline includes integration test step | ||
|
|
||
| ## Key Files | ||
|
|
||
| | File | Purpose | | ||
| |---|---| | ||
| | `src/__integration__/global-setup.ts` | Vitest globalSetup — starts/stops emulate.dev | | ||
| | `src/__integration__/helpers.ts` | Shared test helpers (Vercel SDK + Octokit client factories) | | ||
| | `vitest.config.ts` | Vitest config with unit + integration projects | | ||
| | `emulate.config.yaml` | Seed data for emulate.dev | | ||
| | `src/__integration__/vercel-deployments.test.ts` | Vercel deployment integration tests via SDK | | ||
| | `src/__integration__/vercel-domains.test.ts` | Vercel domain/alias integration tests via SDK | | ||
| | `src/__integration__/github-pr-comments.test.ts` | GitHub PR comment integration tests | | ||
| | `src/__integration__/github-commit-comments.test.ts` | GitHub commit comment integration tests | | ||
| | `.github/workflows/ci.yml` | CI pipeline with integration test job | | ||
|
|
||
| ## Verification | ||
|
|
||
| 1. `pnpm test:unit` — existing unit tests still pass | ||
| 2. `pnpm test:integration` — all integration tests pass | ||
| 3. `pnpm test` — both unit and integration projects pass | ||
| 4. No external network calls during integration tests | ||
| 5. CI workflow runs both unit and integration tests | ||
|
|
||
| ## Progress | ||
|
|
||
| _(Updated by /please:implement)_ | ||
|
|
||
| ## Decision Log | ||
|
|
||
| | Date | Decision | Rationale | | ||
| |---|---|---| | ||
| | 2026-03-26 | Use `@vercel/sdk` with `serverURL` | Type-safe SDK with configurable endpoint; CLI has no API redirect mechanism | | ||
| | 2026-03-26 | Vitest Projects (not separate config) | Single `vitest.config.ts` with `unit` + `integration` projects; cleaner than separate config files | | ||
| | 2026-03-26 | `src/__integration__/` directory | Clear separation from unit tests in `src/__tests__/` | | ||
|
|
||
| ## Surprises & Discoveries | ||
|
|
||
| - Vercel CLI has no `VERCEL_API` env var for custom API endpoints | ||
| - `@vercel/sdk` Zod validation is too strict for emulate.dev responses — switched to direct fetch | ||
| - `@actions/github.getOctokit()` bakes `baseUrl` at module load via `.defaults()` — used `@octokit/rest` directly instead | ||
| - emulate.dev has a programmatic API (`createEmulator`) ideal for Vitest globalSetup | ||
| - emulate.dev v0.2.0 does not support GitHub Deployments API endpoints (404) | ||
|
|
||
| ## Outcomes & Retrospective | ||
|
|
||
| ### What Was Shipped | ||
| - Integration test infrastructure with emulate.dev (Vitest Projects, globalSetup, seed config) | ||
| - Vercel API contract tests (deployments, domains) | ||
| - GitHub API integration tests (PR comments, commit comments) | ||
| - CI pipeline integration (`test:integration` job) | ||
| - GitHub Deployments API tests (gracefully skip until emulate.dev adds support) | ||
|
|
||
| ### What Went Well | ||
| - emulate.dev's programmatic API made Vitest integration seamless | ||
| - Vitest Projects cleanly separated unit and integration concerns | ||
| - Seed config approach keeps tests deterministic | ||
|
|
||
| ### What Could Improve | ||
| - `@vercel/sdk` couldn't be used due to strict Zod validation vs emulator responses — future emulate.dev versions may fix this | ||
| - `@actions/github` defaults pattern required using `@octokit/rest` directly | ||
|
|
||
| ### Tech Debt Created | ||
| - GitHub Deployments API tests are stubs (skip when unsupported) — revisit when emulate.dev adds support | ||
| - Decision log still references `@vercel/sdk` approach that was changed to direct fetch | ||
47 changes: 47 additions & 0 deletions
47
.please/docs/tracks/active/emulate-integration-test-20260326/spec.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Integration Tests with emulate.dev | ||
|
|
||
| > Track: emulate-integration-test-20260326 | ||
|
|
||
| ## Overview | ||
|
|
||
| Add integration tests for vercel-action using [emulate.dev](https://emulate.dev) — a local, stateful API emulator from Vercel Labs that provides production-fidelity Vercel and GitHub API emulation. This replaces the current approach of only having unit tests with mocked dependencies, enabling end-to-end validation of the action's core flows against realistic API responses. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ### Functional Requirements | ||
|
|
||
| - [ ] FR-1: Set up emulate.dev as a dev dependency with Vitest globalSetup to start/stop the emulator | ||
| - [ ] FR-2: Integration tests for the Vercel deployment flow (create deployment, inspect, get URL) | ||
| - [ ] FR-3: Integration tests for alias domain assignment with template variables (`{{PR_NUMBER}}`, `{{BRANCH}}`) | ||
| - [ ] FR-4: Integration tests for GitHub PR/commit comment creation and updates via emulated GitHub API | ||
| - [ ] FR-5: Integration tests for GitHub Deployments API (deployment status creation/updates) | ||
| - [ ] FR-6: Integration tests run in CI via GitHub Actions workflow | ||
|
|
||
| ### Non-functional Requirements | ||
|
|
||
| - [ ] NFR-1: Integration tests complete within 60 seconds | ||
| - [ ] NFR-2: No external network calls — all API interactions go through emulate.dev | ||
| - [ ] NFR-3: Tests are deterministic and isolated (emulator state resets between test suites) | ||
|
|
||
| ## Acceptance Criteria | ||
|
|
||
| - [ ] AC-1: `pnpm test:integration` runs all integration tests using emulated Vercel API (port 4000) and GitHub API (port 4001) | ||
| - [ ] AC-2: Vercel deployment flow test creates a deployment and retrieves a valid deployment URL | ||
| - [ ] AC-3: Alias domain test assigns a custom domain with PR number/branch substitution | ||
| - [ ] AC-4: GitHub comment test creates and updates PR comments with deployment info | ||
| - [ ] AC-5: GitHub Deployments test creates deployment statuses with correct state transitions _(blocked: emulate.dev v0.2.0 does not support GitHub Deployments API; tests skip gracefully until emulator adds support)_ | ||
| - [ ] AC-6: All integration tests pass in CI (added to `.github/workflows/ci.yml`) | ||
| - [ ] AC-7: Code coverage for integration tests is reported separately from unit tests | ||
|
|
||
| ## Out of Scope | ||
|
|
||
| - Replacing existing unit tests (they remain as-is) | ||
| - Testing Vercel CLI command execution (the CLI itself is not emulated; tests target API interactions) | ||
| - Emulating Google API (not used by this action) | ||
| - Performance/load testing | ||
|
|
||
| ## Assumptions | ||
|
|
||
| - emulate.dev (`npx emulate`) supports concurrent Vercel + GitHub API emulation | ||
| - The emulated APIs support the specific endpoint versions used by this action (v13 deployments, v2 user, etc.) | ||
| - emulate.dev can be installed as a dev dependency via npm/pnpm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| tokens: | ||
| test-token: | ||
| login: test-user | ||
| scopes: [repo, user] | ||
|
|
||
| vercel: | ||
| users: | ||
| - username: test-user | ||
| name: Test User | ||
| email: test@example.com | ||
| teams: | ||
| - slug: test-team | ||
| name: Test Team | ||
| projects: | ||
| - name: test-project | ||
| team: test-team | ||
| framework: nextjs | ||
|
|
||
| github: | ||
| users: | ||
| - login: test-user | ||
| name: Test User | ||
| repos: | ||
| - owner: test-user | ||
| name: test-repo | ||
| language: TypeScript |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.