test(integration): add integration tests with emulate.dev #314
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "track_id": "emulate-integration-test-20260326", | ||
| "type": "feature", | ||
| "status": "in_progress", | ||
| "created_at": "2026-03-26T17:20:28+09:00", | ||
| "updated_at": "2026-03-26T17:30:00+09:00", | ||
| "issue": "", | ||
| "pr": "#314" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,176 @@ | ||
| # Plan: Integration Tests with emulate.dev | ||
|
|
||
| > Track: emulate-integration-test-20260326 | ||
| > Spec: [spec.md](./spec.md) | ||
|
|
||
| ## Overview | ||
|
|
||
| - **Source**: /please:plan | ||
| - **Track**: emulate-integration-test-20260326 | ||
| - **Created**: 2026-03-26 | ||
| - **Approach**: Two-layer testing (Vercel SDK + GitHub Octokit) against emulate.dev | ||
|
|
||
| ## Purpose | ||
|
|
||
| Add integration tests using [emulate.dev](https://emulate.dev) — a local, stateful API emulator from Vercel Labs. This enables end-to-end validation of API interactions without hitting production endpoints or requiring network access. | ||
|
|
||
| ## Context | ||
|
|
||
| ### Current State | ||
| - Unit tests use Vitest with `vi.mock()` for all external dependencies | ||
| - No integration tests exist; example workflows serve as manual E2E validation | ||
| - All Vercel interactions go through CLI (`npx vercel`), not direct HTTP | ||
| - GitHub API uses Octokit via `@actions/github.getOctokit()` with hardcoded base URL | ||
|
|
||
| ### Key Constraints | ||
| - emulate.dev provides stateful Vercel API (port 4000) and GitHub API (port 4001) | ||
| - `@vercel/sdk` supports `serverURL` option for custom API endpoints | ||
| - `@actions/github.getOctokit()` supports `baseUrl` option for custom endpoints | ||
|
|
||
| ## Architecture Decision | ||
|
|
||
| **Two test layers against emulate.dev using official SDK clients:** | ||
|
|
||
| 1. **Vercel API integration tests** — Use [`@vercel/sdk`](https://github.com/vercel/sdk) with `serverURL: 'http://localhost:4000'` to test deployment creation, retrieval, and domain/alias management via the type-safe Vercel SDK client. | ||
|
|
||
| 2. **GitHub API integration tests** — Create Octokit client with `baseUrl: http://localhost:4001` and test the action's comment functions (`createCommentOnPullRequest`, `createCommentOnCommit`) against emulate.dev's stateful GitHub API. | ||
|
|
||
| **Why `@vercel/sdk` instead of Vercel CLI?** The CLI has no documented env var to redirect API calls. The SDK's `serverURL` option lets us point directly at emulate.dev, providing type-safe, production-fidelity testing of the same API contracts the CLI uses internally. | ||
|
|
||
| **Why Vitest Projects?** [Vitest Projects](https://vitest.dev/guide/projects) allow defining `unit` and `integration` as separate projects in a single `vitest.config.ts`, with independent `include` patterns, `globalSetup`, and timeouts. This is cleaner than maintaining separate config files. | ||
|
|
||
| ## Tasks | ||
|
|
||
| ### Phase 1: Infrastructure Setup | ||
|
|
||
| - [x] T-1: Install dependencies and create Vitest globalSetup | ||
| - `pnpm add -D emulate @vercel/sdk` | ||
| - Create `src/__integration__/global-setup.ts` using emulate's programmatic API (`createEmulator`) | ||
|
There was a problem hiding this comment. This plan/spec states the integration tests use
Owner
Author
There was a problem hiding this comment. ✅ Applied in 08ae401
Thanks for catching this, @copilot-pull-request-reviewer! |
||
| - Start Vercel (port 4000) and GitHub (port 4001) services in `setup()` | ||
| - Tear down in `teardown()` | ||
| - **Files**: `src/__integration__/global-setup.ts`, `package.json` | ||
| - **Verify**: `pnpm test:integration` starts emulate.dev and exits cleanly | ||
|
|
||
| - [x] T-2: Configure Vitest Projects for unit and integration test separation | ||
| - Refactor `vitest.config.ts` to use [Vitest Projects](https://vitest.dev/guide/projects) | ||
| - Define `unit` project: `include: ['src/__tests__/**/*.test.ts']`, existing coverage thresholds | ||
| - Define `integration` project: `include: ['src/__integration__/**/*.test.ts']`, `globalSetup: 'src/__integration__/global-setup.ts'`, extended timeouts | ||
| - Add `test:unit` and `test:integration` scripts using `vitest run --project unit` / `--project integration` | ||
| - Keep `test` script running all projects (both unit + integration) | ||
| - **Files**: `vitest.config.ts`, `package.json` | ||
| - **Verify**: `pnpm test:integration` runs only integration tests, `pnpm test:unit` runs only unit tests | ||
|
|
||
| - [x] T-3: Create emulate.config.yaml seed data | ||
| - Define tokens, Vercel users/teams/projects, GitHub users/repos | ||
| - Seed data should match action's typical usage (org ID, project ID, repo owner/name) | ||
| - Create shared test helpers (`src/__integration__/helpers.ts`) with Vercel SDK and Octokit client factories | ||
| - **Files**: `emulate.config.yaml`, `src/__integration__/helpers.ts` | ||
| - **Verify**: emulate.dev starts with seed data populated | ||
|
|
||
| ### Phase 2: Vercel API Integration Tests | ||
|
|
||
| - [x] T-4: Test deployment creation and retrieval via Vercel SDK | ||
| - Use `@vercel/sdk` with `serverURL: 'http://localhost:4000'` | ||
| - `vercel.deployments.createDeployment()` — create deployment, verify response structure | ||
| - `vercel.deployments.getDeployment()` — retrieve by ID, verify URL and status fields | ||
| - `vercel.deployments.getDeployments()` — list deployments, verify pagination | ||
| - **Files**: `src/__integration__/vercel-deployments.test.ts` | ||
| - **Verify**: Deployment lifecycle endpoints return expected shapes via SDK | ||
|
|
||
| - [x] T-5: Test domain and alias management via Vercel SDK | ||
| - `vercel.projects.addProjectDomain()` — add domain to project | ||
| - `vercel.projects.getProjectDomains()` — list project domains | ||
| - `vercel.projects.verifyProjectDomain()` — verify domain | ||
| - `vercel.aliases.assignAlias()` — assign alias to deployment | ||
| - **Files**: `src/__integration__/vercel-domains.test.ts` | ||
| - **Verify**: Domain creation, listing, verification, and alias assignment work via SDK | ||
|
|
||
| ### Phase 3: GitHub API Integration Tests | ||
|
|
||
| - [x] T-6: Test PR comment creation and update | ||
| - Create Octokit client with `baseUrl: http://localhost:4001` | ||
| - Test `issues.createComment` → verify comment appears in `issues.listComments` | ||
| - Test `issues.updateComment` → verify comment body is updated | ||
| - Test the find-previous-comment pattern used by the action | ||
| - **Files**: `src/__integration__/github-pr-comments.test.ts` | ||
| - **Verify**: Full PR comment create/find/update cycle works against emulated API | ||
|
|
||
| - [x] T-7: Test commit comment creation and update | ||
| - Test `repos.createCommitComment` → verify in `repos.listCommentsForCommit` | ||
| - Test `repos.updateCommitComment` → verify body update | ||
| - Test the find-previous-comment pattern for push events | ||
| - **Files**: `src/__integration__/github-commit-comments.test.ts` | ||
| - **Verify**: Full commit comment create/find/update cycle works against emulated API | ||
|
|
||
| ### Phase 4: CI Integration | ||
|
|
||
| - [x] T-8: Add integration test job to CI workflow | ||
| - Add `test:integration` job to `.github/workflows/ci.yml` | ||
| - Same Node.js/pnpm setup as existing test job | ||
| - Runs `pnpm test:integration` | ||
| - **Files**: `.github/workflows/ci.yml` | ||
| - **Verify**: CI pipeline includes integration test step | ||
|
|
||
| ## Key Files | ||
|
|
||
| | File | Purpose | | ||
| |---|---| | ||
| | `src/__integration__/global-setup.ts` | Vitest globalSetup — starts/stops emulate.dev | | ||
| | `src/__integration__/helpers.ts` | Shared test helpers (Vercel SDK + Octokit client factories) | | ||
| | `vitest.config.ts` | Vitest config with unit + integration projects | | ||
| | `emulate.config.yaml` | Seed data for emulate.dev | | ||
| | `src/__integration__/vercel-deployments.test.ts` | Vercel deployment integration tests via SDK | | ||
| | `src/__integration__/vercel-domains.test.ts` | Vercel domain/alias integration tests via SDK | | ||
| | `src/__integration__/github-pr-comments.test.ts` | GitHub PR comment integration tests | | ||
| | `src/__integration__/github-commit-comments.test.ts` | GitHub commit comment integration tests | | ||
| | `.github/workflows/ci.yml` | CI pipeline with integration test job | | ||
|
|
||
| ## Verification | ||
|
|
||
| 1. `pnpm test:unit` — existing unit tests still pass | ||
| 2. `pnpm test:integration` — all integration tests pass | ||
| 3. `pnpm test` — both unit and integration projects pass | ||
| 4. No external network calls during integration tests | ||
| 5. CI workflow runs both unit and integration tests | ||
|
|
||
| ## Progress | ||
|
|
||
| _(Updated by /please:implement)_ | ||
|
|
||
| ## Decision Log | ||
|
|
||
| | Date | Decision | Rationale | | ||
| |---|---|---| | ||
| | 2026-03-26 | Use `@vercel/sdk` with `serverURL` | Type-safe SDK with configurable endpoint; CLI has no API redirect mechanism | | ||
| | 2026-03-26 | Vitest Projects (not separate config) | Single `vitest.config.ts` with `unit` + `integration` projects; cleaner than separate config files | | ||
| | 2026-03-26 | `src/__integration__/` directory | Clear separation from unit tests in `src/__tests__/` | | ||
|
|
||
| ## Surprises & Discoveries | ||
|
|
||
| - Vercel CLI has no `VERCEL_API` env var for custom API endpoints | ||
| - `@vercel/sdk` Zod validation is too strict for emulate.dev responses — switched to direct fetch | ||
| - `@actions/github.getOctokit()` bakes `baseUrl` at module load via `.defaults()` — used `@octokit/rest` directly instead | ||
| - emulate.dev has a programmatic API (`createEmulator`) ideal for Vitest globalSetup | ||
| - emulate.dev v0.2.0 does not support GitHub Deployments API endpoints (404) | ||
|
|
||
| ## Outcomes & Retrospective | ||
|
|
||
| ### What Was Shipped | ||
| - Integration test infrastructure with emulate.dev (Vitest Projects, globalSetup, seed config) | ||
| - Vercel API contract tests (deployments, domains) | ||
| - GitHub API integration tests (PR comments, commit comments) | ||
| - CI pipeline integration (`test:integration` job) | ||
| - GitHub Deployments API tests (gracefully skip until emulate.dev adds support) | ||
|
|
||
| ### What Went Well | ||
| - emulate.dev's programmatic API made Vitest integration seamless | ||
| - Vitest Projects cleanly separated unit and integration concerns | ||
| - Seed config approach keeps tests deterministic | ||
|
|
||
| ### What Could Improve | ||
| - `@vercel/sdk` couldn't be used due to strict Zod validation vs emulator responses — future emulate.dev versions may fix this | ||
| - `@actions/github` defaults pattern required using `@octokit/rest` directly | ||
|
|
||
| ### Tech Debt Created | ||
| - GitHub Deployments API tests are stubs (skip when unsupported) — revisit when emulate.dev adds support | ||
| - Decision log still references `@vercel/sdk` approach that was changed to direct fetch | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Integration Tests with emulate.dev | ||
|
|
||
| > Track: emulate-integration-test-20260326 | ||
|
|
||
| ## Overview | ||
|
|
||
| Add integration tests for vercel-action using [emulate.dev](https://emulate.dev) — a local, stateful API emulator from Vercel Labs that provides production-fidelity Vercel and GitHub API emulation. This replaces the current approach of only having unit tests with mocked dependencies, enabling end-to-end validation of the action's core flows against realistic API responses. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ### Functional Requirements | ||
|
|
||
| - [ ] FR-1: Set up emulate.dev as a dev dependency with Vitest globalSetup to start/stop the emulator | ||
| - [ ] FR-2: Integration tests for the Vercel deployment flow (create deployment, inspect, get URL) | ||
| - [ ] FR-3: Integration tests for alias domain assignment with template variables (`{{PR_NUMBER}}`, `{{BRANCH}}`) | ||
| - [ ] FR-4: Integration tests for GitHub PR/commit comment creation and updates via emulated GitHub API | ||
| - [ ] FR-5: Integration tests for GitHub Deployments API (deployment status creation/updates) | ||
| - [ ] FR-6: Integration tests run in CI via GitHub Actions workflow | ||
|
|
||
| ### Non-functional Requirements | ||
|
|
||
| - [ ] NFR-1: Integration tests complete within 60 seconds | ||
| - [ ] NFR-2: No external network calls — all API interactions go through emulate.dev | ||
| - [ ] NFR-3: Tests are deterministic and isolated (emulator state resets between test suites) | ||
|
|
||
| ## Acceptance Criteria | ||
|
|
||
| - [ ] AC-1: `pnpm test:integration` runs all integration tests using emulated Vercel API (port 4000) and GitHub API (port 4001) | ||
| - [ ] AC-2: Vercel deployment flow test creates a deployment and retrieves a valid deployment URL | ||
| - [ ] AC-3: Alias domain test assigns a custom domain with PR number/branch substitution | ||
| - [ ] AC-4: GitHub comment test creates and updates PR comments with deployment info | ||
| - [ ] AC-5: GitHub Deployments test creates deployment statuses with correct state transitions | ||
|
amondnet marked this conversation as resolved.
Outdated
|
||
| - [ ] AC-6: All integration tests pass in CI (added to `.github/workflows/ci.yml`) | ||
| - [ ] AC-7: Code coverage for integration tests is reported separately from unit tests | ||
|
|
||
| ## Out of Scope | ||
|
|
||
| - Replacing existing unit tests (they remain as-is) | ||
| - Testing Vercel CLI command execution (the CLI itself is not emulated; tests target API interactions) | ||
| - Emulating Google API (not used by this action) | ||
| - Performance/load testing | ||
|
|
||
| ## Assumptions | ||
|
|
||
| - emulate.dev (`npx emulate`) supports concurrent Vercel + GitHub API emulation | ||
| - The emulated APIs support the specific endpoint versions used by this action (v13 deployments, v2 user, etc.) | ||
| - emulate.dev can be installed as a dev dependency via npm/pnpm | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| tokens: | ||
| test-token: | ||
| login: test-user | ||
| scopes: [repo, user] | ||
|
|
||
| vercel: | ||
| users: | ||
| - username: test-user | ||
| name: Test User | ||
| email: test@example.com | ||
| teams: | ||
| - slug: test-team | ||
| name: Test Team | ||
| projects: | ||
| - name: test-project | ||
| team: test-team | ||
| framework: nextjs | ||
|
|
||
| github: | ||
| users: | ||
| - login: test-user | ||
| name: Test User | ||
| repos: | ||
| - owner: test-user | ||
| name: test-repo | ||
| language: TypeScript |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
testjob runspnpm test, which (with Vitest Projects) will execute both unit and integration suites. This makes the newtest-integrationjob redundant and increases CI time. Consider changing thetestjob to runpnpm test:unit(and keep integration in the dedicated job), or remove the separate integration job if you wantpnpm testto remain the single source of truth.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Applied in 08ae401
.github/workflows/ci.ymltestjob to runpnpm test:unitso integration tests are not executed twice (unit job + integration job)Thanks for the suggestion, @copilot-pull-request-reviewer!