|
| 1 | +# Monorepo testing overview |
| 2 | + |
| 3 | +This document maps **how tests are organized**, **which commands run what**, and **how CI fits in**. It is aimed at humans who want a single mental model—and at planning simplifications. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Two different “default” test flows |
| 8 | + |
| 9 | +The repo effectively has **two parallel ways** to run automated tests: |
| 10 | + |
| 11 | +| Flow | Command | Mechanism | Typical use | |
| 12 | +| --------------------- | --------------------- | ------------------------------------------------------- | ------------------------------------------- | |
| 13 | +| **CI / coverage** | `yarn test` | Root `vitest.config.ts` with `projects: ['packages/*']` | GitHub Actions “Test” job, local coverage | |
| 14 | +| **Developer / Turbo** | `yarn test:dev:quiet` | `turbo run test:dev:quiet` (depends on `^build`) | Fast feedback, per-package `test:dev:quiet` | |
| 15 | + |
| 16 | +They are **not** the same: |
| 17 | + |
| 18 | +- **`yarn test`** runs Vitest once at the repo root and discovers **all** `packages/*` as Vitest projects (each package’s own `vitest.config.ts` still applies as a **project** merge). |
| 19 | +- **`yarn test:dev:quiet`** runs **only** workspaces that define `test:dev:quiet`, after **building dependencies** via Turborepo (`turbo.json` lists `test:dev` / `test:dev:quiet` with `dependsOn: ["^build"]`). |
| 20 | + |
| 21 | +**Implication:** A package can pass in one flow and behave differently in the other if build artifacts or config merging differ. Prefer documenting “CI runs X, local fast loop runs Y” explicitly. |
| 22 | + |
| 23 | +--- |
| 24 | + |
| 25 | +## Root scripts (cheat sheet) |
| 26 | + |
| 27 | +| Script | What it does | |
| 28 | +| ----------------------- | --------------------------------------------------------------------------------------------------------- | |
| 29 | +| `yarn test` | Root Vitest, all workspace projects; coverage enabled per root config | |
| 30 | +| `yarn test:dev` | Turbo → each package’s `test:dev` | |
| 31 | +| `yarn test:dev:quiet` | Turbo → each package’s `test:dev:quiet` (silent reporter) | |
| 32 | +| `yarn test:integration` | `yarn workspaces foreach --all run test:integration` — **only workspaces that define** `test:integration` | |
| 33 | +| `yarn test:e2e` | `foreach --all run test:e2e` — only workspaces that define `test:e2e` | |
| 34 | +| `yarn test:e2e:ci` | `foreach --all run test:e2e:ci` — CI-oriented entrypoints per package | |
| 35 | +| `yarn test:e2e:local` | `foreach --all run test:e2e:local` — today essentially **`@ocap/kernel-test-local` only** | |
| 36 | + |
| 37 | +Yarn **skips** workspaces that do not define the script (no failure). |
| 38 | + |
| 39 | +--- |
| 40 | + |
| 41 | +## Vitest layout (global) |
| 42 | + |
| 43 | +- **Root** [`vitest.config.ts`](../vitest.config.ts): `projects: ['packages/*']`, `pool: 'threads'`, default `testTimeout: 2000`, **global** `setupFiles` including **`vitest-fetch-mock`** (via `@ocap/repo-tools/test-utils/fetch-mock`). |
| 44 | +- **Packages** usually use `@ocap/repo-tools/vitest-config` **`mergeConfig`** with the root config and `defineProject({ test: { name: '…' } })` to add package-specific `setupFiles`, `include`/`exclude`, timeouts, etc. |
| 45 | + |
| 46 | +### Global setup footguns |
| 47 | + |
| 48 | +1. **`vitest-fetch-mock`** (root) — Mocks **`fetch`**. Any test that needs a **real** HTTP client to `localhost` (Docker Anvil, local servers) will see empty or wrong bodies unless the test project **opts out** (separate Vitest config with `setupFiles: []`, or exclude those files from the default project). **`@ocap/evm-wallet-experiment`** Docker E2E is an example: excluded from the main project; run with `yarn test:e2e:docker` and `vitest.config.docker.ts`. |
| 49 | +2. **`endoify` vs `mock-endoify`** — Kernel-ish packages pick different setup files (`@metamask/kernel-shims/endoify-node`, `mock-endoify`, or real `endoify.js` for some E2E). Mixing these affects lockdown / SES behavior. |
| 50 | +3. **`testTimeout: 2000`** at root — Fine for unit tests; packages that need longer runs override in their project (e.g. kernel tests, E2E configs). |
| 51 | + |
| 52 | +--- |
| 53 | + |
| 54 | +## Categories of tests (vocabulary) |
| 55 | + |
| 56 | +| Term in this repo | Meaning | Examples | |
| 57 | +| -------------------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | |
| 58 | +| **Unit** | Vitest, co-located `*.test.ts`, no external services | Most `packages/*/src/**/*.test.ts` | |
| 59 | +| **Integration** (script: `test:integration`) | Separate Vitest config, often `pool: 'forks'`, different `setupFiles` / env | `kernel-cli`, `kernel-browser-runtime`, `evm-wallet-experiment` (`test/integration/**`) | |
| 60 | +| **E2E** (script: `test:e2e`) | **Not one technology**: Vitest extra config, **Playwright**, or **Node drivers** | See below | |
| 61 | +| **Docker E2E** | Vitest against **Docker Compose** stack (Anvil, bundler, kernels) | `evm-wallet-experiment` → `yarn test:e2e:docker` | |
| 62 | +| **Local E2E** | Vitest, **external LMS** (Ollama / llama-cpp), not in default CI | `kernel-test-local` → `yarn test:e2e:local` | |
| 63 | +| **KLMS E2E** | Vitest hitting real **Ollama** / API | `kernel-language-model-service` → `yarn test:e2e` in that package | |
| 64 | + |
| 65 | +--- |
| 66 | + |
| 67 | +## Packages with non-standard test entrypoints |
| 68 | + |
| 69 | +### `@ocap/kernel-test` |
| 70 | + |
| 71 | +- **Role:** Kernel tests that **spin up / talk to vats** (see package description). |
| 72 | +- **Scripts:** Standard `test` / `test:dev` / `test:dev:quiet` only. |
| 73 | +- **Config:** `endoify-node` setup, longer `testTimeout`, coverage thresholds unless `development` mode. |
| 74 | + |
| 75 | +### `@ocap/kernel-test-local` |
| 76 | + |
| 77 | +- **Role:** **Local-only** flows (e.g. agents + **Ollama**); not assumed available in CI. |
| 78 | +- **Unit-ish Vitest:** `vitest.config.ts` — `include: ./src/**/*.test.ts`, **`exclude: **/\*.e2e.test.ts`\*\*. |
| 79 | +- **Local E2E:** `test:e2e:local` → `vitest.config.e2e.ts` — includes `*.e2e.test.ts` and `./test/**/*.test.ts`, long timeouts. |
| 80 | +- **Helpers:** `test:e2e:local:llama-cpp` sets `LMS_PROVIDER=llama-cpp`. |
| 81 | + |
| 82 | +### `@ocap/evm-wallet-experiment` |
| 83 | + |
| 84 | +| Script | Purpose | |
| 85 | +| -------------------------- | -------------------------------------------------------------------------------------------------------- | |
| 86 | +| `test` / `test:dev*` | Main Vitest; **`test/integration/**` excluded**; **`test/e2e/docker/**` excluded** (fetch mock conflict) | |
| 87 | +| `test:integration` | `vitest.config.integration.ts` → `test/integration/**` | |
| 88 | +| `test:e2e:docker` | `vitest.config.docker.ts` — Docker Compose E2E, **no fetch mock**, `pool: 'forks'` | |
| 89 | +| `test:e2e` / `test:e2e:ci` | **Node** driver: `test/e2e/run-all-e2e.mjs` (after `yarn build`) | |
| 90 | +| `test:node`, `test:node:*` | Additional **Node** integration/e2e scripts (wallet, peer, Sepolia, etc.) | |
| 91 | + |
| 92 | +CI **E2E matrix** runs `yarn workspace @ocap/evm-wallet-experiment test:e2e:ci` (the Node runner), **not** `test:e2e:docker`—Docker Compose is **manual / optional** unless added to CI. |
| 93 | + |
| 94 | +### `@ocap/kernel-language-model-service` |
| 95 | + |
| 96 | +- **Unit:** default `vitest.config.ts`. |
| 97 | +- **`test:e2e`:** `vitest.config.e2e.ts` — `include: test/e2e/**`, real **endoify** setup; hits real services (e.g. Ollama). |
| 98 | + |
| 99 | +### `@metamask/kernel-node-runtime` |
| 100 | + |
| 101 | +- **`test:e2e`:** Vitest `vitest.config.e2e.ts`. |
| 102 | +- **`test:e2e:ci`:** Shell script: bundle + serve test vats, then `yarn test:e2e`. |
| 103 | + |
| 104 | +### `@ocap/kernel-cli` |
| 105 | + |
| 106 | +- **`test:e2e` / `test:integration`:** Separate Vitest config files (`vitest.e2e.config.ts`, `vitest.integration.config.ts`). |
| 107 | +- **`test:e2e:ci`:** Same as `test:e2e`. |
| 108 | + |
| 109 | +### `@metamask/kernel-browser-runtime` |
| 110 | + |
| 111 | +- **`test:integration`:** `vitest.integration.config.ts` (browser-oriented integration). |
| 112 | + |
| 113 | +### `@ocap/extension` and `@ocap/omnium-gatherum` |
| 114 | + |
| 115 | +- **`test:e2e`:** **Playwright** (`yarn playwright test`). |
| 116 | +- **`test:e2e:ci`:** Extension uses a **wrapper script** (build, bundle kernel-test vats, serve, then Playwright). Omnium uses `yarn test:e2e` directly. |
| 117 | + |
| 118 | +### `@ocap/omnium-gatherum` / `@ocap/extension` (extra) |
| 119 | + |
| 120 | +- **`test:build`:** TSX build tests (distinct from Vitest unit tests). |
| 121 | + |
| 122 | +--- |
| 123 | + |
| 124 | +## CI (`.github/workflows/lint-build-test.yml`) |
| 125 | + |
| 126 | +Rough pipeline: |
| 127 | + |
| 128 | +1. **Lint** — `yarn lint` |
| 129 | +2. **Build** — `yarn build` |
| 130 | +3. **Test** — `yarn build` then **`yarn test --coverage=…`** (root Vitest + projects) |
| 131 | +4. **Integration** — `yarn build` then **`yarn test:integration`** (only packages with that script) |
| 132 | +5. **E2E** — matrix on Node 24, `VITE_DB_FOLDER=e2e yarn build`, then **`yarn workspace <pkg> test:e2e:ci`** for: |
| 133 | + - `@metamask/kernel-node-runtime` |
| 134 | + - `@ocap/extension` |
| 135 | + - `@ocap/omnium-gatherum` |
| 136 | + - `@ocap/evm-wallet-experiment` |
| 137 | + |
| 138 | +**Not run in that workflow by default:** |
| 139 | + |
| 140 | +- **`yarn test:dev` / `yarn test:dev:quiet`** (Turbo) |
| 141 | +- **`kernel-test-local`** `test:e2e:local` (needs local Ollama / llama-cpp) |
| 142 | +- **`evm-wallet-experiment`** `test:e2e:docker` (needs Docker Compose stack) |
| 143 | +- **`kernel-language-model-service`** `test:e2e` against live Ollama (unless added elsewhere) |
| 144 | +- **`@metamask/kernel-cli`** `test:e2e` / `test:e2e:ci` — defined on the package but **not** in the E2E matrix; only runs if you invoke the workspace script or a future `yarn test:e2e` full foreach locally |
| 145 | + |
| 146 | +Playwright browsers: **playwright-install** action is used for the Test and E2E jobs. |
| 147 | + |
| 148 | +--- |
| 149 | + |
| 150 | +## Turborepo |
| 151 | + |
| 152 | +[`turbo.json`](../turbo.json) only wires **`build`**, **`build:dev`**, **`test:dev`**, **`test:dev:quiet`**. Other scripts (`test`, `test:integration`, `test:e2e*`) are **not** Turbo tasks—run them via root `yarn …` or per workspace. |
| 153 | + |
| 154 | +--- |
| 155 | + |
| 156 | +## Simplification ideas (for discussion) |
| 157 | + |
| 158 | +1. **Name alignment** — Today “E2E” means Vitest, Playwright, Node scripts, or Docker. Consider prefixes in docs/scripts: `test:e2e:vitest`, `test:e2e:playwright`, `test:e2e:node`, `test:e2e:docker` (even if kept as aliases). |
| 159 | +2. **Single “full local checklist” doc** — One section: “Before push: `yarn lint`, `yarn build`, `yarn test`, `yarn test:integration`”; optional: “With Docker: …”, “With Ollama: …”. |
| 160 | +3. **CI vs dev parity** — Decide whether **`yarn test:dev:quiet`** should mirror CI more closely (e.g. document that CI uses **`yarn test`**, not Turbo). |
| 161 | +4. **Fetch mock** — Document in contributor guide: “Tests that need real `fetch` must not inherit root `setupFiles`” (pattern: separate config + dedicated script, as with Docker E2E). |
| 162 | +5. **Consolidate `test:e2e` vs `test:e2e:ci`** — Where they are identical (e.g. omnium, kernel-cli), one script could delegate to the other to reduce duplication; where they differ (extension, kernel-node-runtime), keep both and document why. |
| 163 | +6. **Optional CI job** — `evm-wallet` Docker E2E as a manual or nightly workflow with compose, if the team wants regression coverage without slowing every PR. |
| 164 | +7. **`kernel-test` vs `kernel-test-local`** — Names are good; a one-line diagram in README (“kernel-test = in-process vats; kernel-test-local = needs Ollama”) helps onboarding. |
| 165 | + |
| 166 | +--- |
| 167 | + |
| 168 | +## File index (quick) |
| 169 | + |
| 170 | +| Area | Config / driver | |
| 171 | +| ----------------------- | ------------------------------------------------------------- | |
| 172 | +| Root Vitest | `vitest.config.ts` | |
| 173 | +| Turbo | `turbo.json` | |
| 174 | +| Docker E2E (evm-wallet) | `packages/evm-wallet-experiment/vitest.config.docker.ts` | |
| 175 | +| Local agent E2E | `packages/kernel-test-local/vitest.config.e2e.ts` | |
| 176 | +| KLMS service E2E | `packages/kernel-language-model-service/vitest.config.e2e.ts` | |
| 177 | +| evm-wallet integration | `packages/evm-wallet-experiment/vitest.config.integration.ts` | |
| 178 | +| Fetch mock setup | `packages/repo-tools/src/test-utils/env/fetch-mock.ts` | |
| 179 | + |
| 180 | +--- |
| 181 | + |
| 182 | +_Generated for navigation and planning; update when scripts or CI change._ |
0 commit comments