-
-
Notifications
You must be signed in to change notification settings - Fork 116
test(integration): add integration tests with emulate.dev #314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
b5def0f
05982d3
99a08db
764a0f2
c03ba8b
3f27ec5
8f33a3a
074a8e2
beb36b8
c938bc7
3e10052
82a6c24
2e7b21e
840e778
889b3f2
198f169
08ae401
b9ae17e
021a335
0fd42e4
bd4d1ac
e2309f3
ff1e1b3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "track_id": "emulate-integration-test-20260326", | ||
| "type": "feature", | ||
| "status": "in_progress", | ||
| "created_at": "2026-03-26T17:20:28+09:00", | ||
| "updated_at": "2026-03-26T17:30:00+09:00", | ||
| "issue": "", | ||
| "pr": "#314" | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,173 @@ | ||
| # Plan: Integration Tests with emulate.dev | ||
|
|
||
| > Track: emulate-integration-test-20260326 | ||
| > Spec: [spec.md](./spec.md) | ||
|
|
||
| ## Overview | ||
|
|
||
| - **Source**: /please:plan | ||
| - **Track**: emulate-integration-test-20260326 | ||
| - **Created**: 2026-03-26 | ||
| - **Approach**: Two-layer testing (direct fetch for Vercel API + GitHub Octokit) against emulate.dev | ||
|
|
||
| ## Purpose | ||
|
|
||
| Add integration tests using [emulate.dev](https://emulate.dev) — a local, stateful API emulator from Vercel Labs. This enables end-to-end validation of API interactions without hitting production endpoints or requiring network access. | ||
|
|
||
| ## Context | ||
|
|
||
| ### Current State | ||
| - Unit tests use Vitest with `vi.mock()` for all external dependencies | ||
| - No integration tests exist; example workflows serve as manual E2E validation | ||
| - All Vercel interactions go through CLI (`npx vercel`), not direct HTTP | ||
| - GitHub API uses Octokit via `@actions/github.getOctokit()` with hardcoded base URL | ||
|
|
||
| ### Key Constraints | ||
| - emulate.dev provides stateful Vercel API (port 4000) and GitHub API (port 4001) | ||
| - Direct `fetch` with `Authorization: Bearer <token>` is used for Vercel API calls | ||
| - `@actions/github.getOctokit()` bakes `baseUrl` at load time — `@octokit/rest` used directly with `baseUrl` option | ||
|
|
||
| ## Architecture Decision | ||
|
|
||
| **Two test layers against emulate.dev:** | ||
|
|
||
| 1. **Vercel API integration tests** — Use direct `fetch` calls with `Authorization: Bearer <token>` headers against `http://localhost:4000` to test deployment creation, retrieval, and domain/alias management. (`@vercel/sdk` was evaluated but dropped due to strict Zod validation that rejects emulate.dev responses.) | ||
|
|
||
| 2. **GitHub API integration tests** — Create Octokit client with `baseUrl: http://localhost:4001` and test the action's comment functions (`createCommentOnPullRequest`, `createCommentOnCommit`) against emulate.dev's stateful GitHub API. | ||
|
|
||
| **Why direct fetch instead of `@vercel/sdk`?** The SDK's Zod validation layer is too strict for emulate.dev's responses — it rejects fields the emulator omits or approximates. Direct `fetch` with manual JSON parsing provides the same coverage without validation friction. | ||
|
|
||
| **Why Vitest Projects?** [Vitest Projects](https://vitest.dev/guide/projects) allow defining `unit` and `integration` as separate projects in a single `vitest.config.ts`, with independent `include` patterns, `globalSetup`, and timeouts. This is cleaner than maintaining separate config files. | ||
|
|
||
| ## Tasks | ||
|
|
||
| ### Phase 1: Infrastructure Setup | ||
|
|
||
| - [x] T-1: Install dependencies and create Vitest globalSetup | ||
| - `pnpm add -D emulate` (note: `@vercel/sdk` was evaluated but dropped — see Architecture Decision section) | ||
| - Create `src/__integration__/global-setup.ts` using emulate's programmatic API (`createEmulator`) | ||
| - Start Vercel (port 4000) and GitHub (port 4001) services in `setup()` | ||
| - Tear down in `teardown()` | ||
| - **Files**: `src/__integration__/global-setup.ts`, `package.json` | ||
| - **Verify**: `pnpm test:integration` starts emulate.dev and exits cleanly | ||
|
|
||
| - [x] T-2: Configure Vitest Projects for unit and integration test separation | ||
| - Refactor `vitest.config.ts` to use [Vitest Projects](https://vitest.dev/guide/projects) | ||
| - Define `unit` project: `include: ['src/__tests__/**/*.test.ts']`, existing coverage thresholds | ||
| - Define `integration` project: `include: ['src/__integration__/**/*.test.ts']`, `globalSetup: 'src/__integration__/global-setup.ts'`, extended timeouts | ||
| - Add `test:unit` and `test:integration` scripts using `vitest run --project unit` / `--project integration` | ||
| - Keep `test` script running all projects (both unit + integration) | ||
| - **Files**: `vitest.config.ts`, `package.json` | ||
| - **Verify**: `pnpm test:integration` runs only integration tests, `pnpm test:unit` runs only unit tests | ||
|
|
||
| - [x] T-3: Create emulate.config.yaml seed data | ||
| - Define tokens, Vercel users/teams/projects, GitHub users/repos | ||
| - Seed data should match action's typical usage (org ID, project ID, repo owner/name) | ||
| - Create shared test helpers (`src/__integration__/helpers.ts`) with `vercelFetch` and `createOctokitClient` factories | ||
| - **Files**: `emulate.config.yaml`, `src/__integration__/helpers.ts` | ||
| - **Verify**: emulate.dev starts with seed data populated | ||
|
|
||
| ### Phase 2: Vercel API Integration Tests | ||
|
|
||
| - [x] T-4: Test deployment creation and retrieval via direct fetch | ||
| - Use direct `fetch` with `Authorization: Bearer <token>` against `http://localhost:4000` | ||
| - `POST /v13/deployments` — create deployment, verify response structure | ||
| - `GET /v13/deployments/:id` — retrieve by ID via `VercelApiClient.inspect()` | ||
| - **Files**: `src/__integration__/vercel-api.test.ts` | ||
| - **Verify**: Deployment lifecycle endpoints return expected shapes | ||
|
|
||
| - [x] T-5: Test domain and alias management via direct fetch | ||
| - `POST /v9/projects/:id/domains` — add domain to project | ||
| - `GET /v9/projects/:id/domains` — list project domains | ||
| - `POST /v9/projects/:id/domains/:domain/verify` — verify domain | ||
| - `POST /v2/deployments/:id/aliases` — assign alias (skips gracefully if emulator returns 404) | ||
| - **Files**: `src/__integration__/vercel-api.test.ts` | ||
| - **Verify**: Domain creation, listing, verification, and alias assignment work via direct fetch | ||
|
|
||
| ### Phase 3: GitHub API Integration Tests | ||
|
|
||
| - [x] T-6: Test PR comment creation and update | ||
| - Create Octokit client with `baseUrl: http://localhost:4001` | ||
| - Test `issues.createComment` → verify comment appears in `issues.listComments` | ||
| - Test `issues.updateComment` → verify comment body is updated | ||
| - Test the find-previous-comment pattern used by the action | ||
| - **Files**: `src/__integration__/github-pr-comments.test.ts` | ||
| - **Verify**: Full PR comment create/find/update cycle works against emulated API | ||
|
|
||
| - [x] T-7: Test commit comment creation and update | ||
| - Test `repos.createCommitComment` → verify in `repos.listCommentsForCommit` | ||
| - Test `repos.updateCommitComment` → verify body update | ||
| - Test the find-previous-comment pattern for push events | ||
| - **Files**: `src/__integration__/github-commit-comments.test.ts` | ||
| - **Verify**: Full commit comment create/find/update cycle works against emulated API | ||
|
|
||
| ### Phase 4: CI Integration | ||
|
|
||
| - [x] T-8: Add integration test job to CI workflow | ||
| - Add `test:integration` job to `.github/workflows/ci.yml` | ||
| - Same Node.js/pnpm setup as existing test job | ||
| - Runs `pnpm test:integration` | ||
| - **Files**: `.github/workflows/ci.yml` | ||
| - **Verify**: CI pipeline includes integration test step | ||
|
|
||
| ## Key Files | ||
|
|
||
| | File | Purpose | | ||
| |---|---| | ||
| | `src/__integration__/global-setup.ts` | Vitest globalSetup — starts/stops emulate.dev | | ||
| | `src/__integration__/helpers.ts` | Shared test helpers (`vercelFetch` + `createOctokitClient` factories) | | ||
| | `vitest.config.ts` | Vitest config with unit + integration projects | | ||
| | `emulate.config.yaml` | Seed data for emulate.dev | | ||
| | `src/__integration__/vercel-api.test.ts` | Vercel deployment, domain, and alias integration tests via direct fetch | | ||
| | `src/__integration__/github-pr-comments.test.ts` | GitHub PR comment integration tests | | ||
| | `src/__integration__/github-commit-comments.test.ts` | GitHub commit comment integration tests | | ||
| | `.github/workflows/ci.yml` | CI pipeline with integration test job | | ||
|
|
||
| ## Verification | ||
|
|
||
| 1. `pnpm test:unit` — existing unit tests still pass | ||
| 2. `pnpm test:integration` — all integration tests pass | ||
| 3. `pnpm test` — both unit and integration projects pass | ||
| 4. No external network calls during integration tests | ||
| 5. CI workflow runs both unit and integration tests | ||
|
|
||
| ## Progress | ||
|
|
||
| _(Updated by /please:implement)_ | ||
|
|
||
| ## Decision Log | ||
|
|
||
| | Date | Decision | Rationale | | ||
| |---|---|---| | ||
| | 2026-03-26 | Use direct `fetch` for Vercel API | `@vercel/sdk` evaluated but dropped — Zod validation too strict for emulate.dev responses; direct fetch avoids this friction | | ||
| | 2026-03-26 | Vitest Projects (not separate config) | Single `vitest.config.ts` with `unit` + `integration` projects; cleaner than separate config files | | ||
| | 2026-03-26 | `src/__integration__/` directory | Clear separation from unit tests in `src/__tests__/` | | ||
|
|
||
| ## Surprises & Discoveries | ||
|
|
||
| - Vercel CLI has no `VERCEL_API` env var for custom API endpoints | ||
| - `@vercel/sdk` Zod validation is too strict for emulate.dev responses — switched to direct fetch | ||
| - `@actions/github.getOctokit()` bakes `baseUrl` at module load via `.defaults()` — used `@octokit/rest` directly instead | ||
| - emulate.dev has a programmatic API (`createEmulator`) ideal for Vitest globalSetup | ||
| - emulate.dev v0.2.0 does not support GitHub Deployments API endpoints (404) | ||
|
|
||
| ## Outcomes & Retrospective | ||
|
|
||
| ### What Was Shipped | ||
| - Integration test infrastructure with emulate.dev (Vitest Projects, globalSetup, seed config) | ||
| - Vercel API contract tests (deployments, domains) | ||
| - GitHub API integration tests (PR comments, commit comments) | ||
| - CI pipeline integration (`test:integration` job) | ||
| - GitHub Deployments API tests (gracefully skip until emulate.dev adds support) | ||
|
|
||
| ### What Went Well | ||
| - emulate.dev's programmatic API made Vitest integration seamless | ||
| - Vitest Projects cleanly separated unit and integration concerns | ||
| - Seed config approach keeps tests deterministic | ||
|
|
||
| ### What Could Improve | ||
| - `@vercel/sdk` couldn't be used due to strict Zod validation vs emulator responses — future emulate.dev versions may fix this | ||
| - `@actions/github` defaults pattern required using `@octokit/rest` directly | ||
|
|
||
| ### Tech Debt Created | ||
| - GitHub Deployments API tests are stubs (skip when unsupported) — revisit when emulate.dev adds support | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Integration Tests with emulate.dev | ||
|
|
||
| > Track: emulate-integration-test-20260326 | ||
|
|
||
| ## Overview | ||
|
|
||
| Add integration tests for vercel-action using [emulate.dev](https://emulate.dev) — a local, stateful API emulator from Vercel Labs that provides production-fidelity Vercel and GitHub API emulation. This replaces the current approach of only having unit tests with mocked dependencies, enabling end-to-end validation of the action's core flows against realistic API responses. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ### Functional Requirements | ||
|
|
||
| - [ ] FR-1: Set up emulate.dev as a dev dependency with Vitest globalSetup to start/stop the emulator | ||
| - [ ] FR-2: Integration tests for the Vercel deployment flow (create deployment, inspect, get URL) | ||
| - [ ] FR-3: Integration tests for alias domain assignment with template variables (`{{PR_NUMBER}}`, `{{BRANCH}}`) | ||
| - [ ] FR-4: Integration tests for GitHub PR/commit comment creation and updates via emulated GitHub API | ||
| - [ ] FR-5: Integration tests for GitHub Deployments API (deployment status creation/updates) | ||
| - [ ] FR-6: Integration tests run in CI via GitHub Actions workflow | ||
|
|
||
| ### Non-functional Requirements | ||
|
|
||
| - [ ] NFR-1: Integration tests complete within 60 seconds | ||
| - [ ] NFR-2: No external network calls — all API interactions go through emulate.dev | ||
| - [ ] NFR-3: Tests are deterministic and isolated (emulator state resets between test suites) | ||
|
|
||
| ## Acceptance Criteria | ||
|
|
||
| - [ ] AC-1: `pnpm test:integration` runs all integration tests using emulated Vercel API (port 4000) and GitHub API (port 4001) | ||
| - [ ] AC-2: Vercel deployment flow test creates a deployment and retrieves a valid deployment URL | ||
| - [ ] AC-3: Alias domain test assigns a custom domain with PR number/branch substitution | ||
| - [ ] AC-4: GitHub comment test creates and updates PR comments with deployment info | ||
| - [ ] AC-5: GitHub Deployments test creates deployment statuses with correct state transitions _(blocked: emulate.dev v0.2.0 does not support GitHub Deployments API; tests skip gracefully until emulator adds support)_ | ||
| - [ ] AC-6: All integration tests pass in CI (added to `.github/workflows/ci.yml`) | ||
| - [ ] AC-7: Code coverage for integration tests is reported separately from unit tests | ||
|
|
||
| ## Out of Scope | ||
|
|
||
| - Replacing existing unit tests (they remain as-is) | ||
| - Testing Vercel CLI command execution (the CLI itself is not emulated; tests target API interactions) | ||
| - Emulating Google API (not used by this action) | ||
| - Performance/load testing | ||
|
|
||
| ## Assumptions | ||
|
|
||
| - emulate.dev (`npx emulate`) supports concurrent Vercel + GitHub API emulation | ||
| - The emulated APIs support the specific endpoint versions used by this action (v13 deployments, v2 user, etc.) | ||
| - emulate.dev can be installed as a dev dependency via npm/pnpm |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1 +1,7 @@ | ||||||
| export default { extends: ['@commitlint/config-conventional'] } | ||||||
| export default { | ||||||
| extends: ['@commitlint/config-conventional'], | ||||||
| rules: { | ||||||
| 'body-max-line-length': [2, 'always', 200], | ||||||
| 'footer-max-line-length': [2, 'always', 200], | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Revert footer max line length to 100 to align with the repo’s documented commit convention. (Based on your team's feedback about enforcing 100-character commitlint limits.) Prompt for AI agents
Suggested change
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⏭️ Not applied
Thank you for the suggestion, @cubic-dev-ai! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Got it—thanks for clarifying. I’ll treat 200-char body/footer limits as correct for this repo going forward. |
||||||
| }, | ||||||
| } | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| export declare function setup(): Promise<void>; | ||
| export declare function teardown(): Promise<void>; |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| import { Octokit } from '@octokit/rest'; | ||
| export declare const VERCEL_TOKEN = "test-token"; | ||
| export declare const GITHUB_TOKEN = "test-token"; | ||
| export declare const TEST_OWNER = "test-user"; | ||
| export declare const TEST_REPO = "test-repo"; | ||
| export declare const TEST_TEAM = "test-team"; | ||
| export declare const TEST_PROJECT = "test-project"; | ||
| export declare function vercelFetch(path: string, options?: RequestInit): Promise<Response>; | ||
| export declare function createOctokitClient(): Octokit; |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,3 @@ | ||
| import type { ActionConfig, OctokitClient } from './types'; | ||
| export declare function createCommentOnCommit(octokit: OctokitClient, config: ActionConfig, deploymentCommit: string, deploymentUrl: string, deploymentName: string): Promise<void>; | ||
| export declare function createCommentOnPullRequest(octokit: OctokitClient, config: ActionConfig, deploymentCommit: string, deploymentUrl: string, deploymentName: string): Promise<void>; | ||
| import type { ActionConfig, GitHubContext, OctokitClient } from './types'; | ||
| export declare function createCommentOnCommit(octokit: OctokitClient, ctx: GitHubContext, config: ActionConfig, deploymentCommit: string, deploymentUrl: string, deploymentName: string): Promise<void>; | ||
| export declare function createCommentOnPullRequest(octokit: OctokitClient, ctx: GitHubContext, config: ActionConfig, deploymentCommit: string, deploymentUrl: string, deploymentName: string): Promise<void>; |
Uh oh!
There was an error while loading. Please reload this page.