feat: integration mount evals + prescriptive spawn instructions for non-claude CLIs (#375)

* feat: integration mount evals + prescriptive spawn instructions for non-claude CLIs

Adds an eval harness (evals/) that spawns real agents via the broker in a
fixture dir with a fake .integrations/ mount, then scores whether the agent
wrote to the correct path with a valid JSON payload. Covers 6 scenarios
(Slack channel/DM, Linear create/update/comment/delete) across 5 guidance
variants (bare, claude-md, slim-inject, full-inject, prescriptive).

Key findings from eval runs:
- `prescriptive` variant achieves 18/18 (100%) across all free and Chinese
  models (deepseek-v4-flash-free, mimo, nemotron, north-mini-code, gpt-5.4-nano,
  gpt-5.4-mini, gpt-5.1-codex-mini, gpt-5.5) — reliable for non-claude CLIs
- `full-inject` and `slim-inject` also reach 100% once absolute paths are
  injected for CLIs whose cwd doesn't match the project fixture dir
- `bare` fails universally — no model self-discovers integration paths

Harness changes:
- opencode uses `spawnCli({ transport: 'headless' })` + `skipRelayPrompt`
- Default opencode model is `opencode/deepseek-v4-flash-free` (free, fast)
- Non-claude CLIs receive absolute fixture paths in the task prefix so writes
  land in the correct temp dir regardless of CLI cwd detection

Production wiring:
- `IntegrationsManager.prescriptiveSpawnInstructions()`: derives the lookup
  table from real `writebackCommandMountPaths` — same data as
  `initialSpawnInstructions`, compact format instead of narrative prose
- `broker:spawn-agent` IPC handler routes `cli !== 'claude'` to prescriptive;
  `recordSpawnInstructionDelivery` guarded to narrative path only

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: CI test mock + address PR review feedback

CI fix:
- ipc-handlers.test.ts mocked integrationsManager was missing
  prescriptiveSpawnInstructions/recordSpawnInstructionDelivery, so the
  broker:spawn-agent test (cli: 'codex') threw "not a function". Added the
  mocks plus focused routing tests (non-claude → prescriptive + no delivery
  record; claude → narrative + delivery record).

Review feedback:
- package.json: depend on published @agent-relay/evals@^8.8.2 instead of
  file:../relay/packages/evals so fresh clones / CI can resolve it (codex P2)
- integrations.ts: Linear comment path now references the canonical issue
  resource file (<KEY>-<num>__<uuid>.json) instead of a bare <issueId> dir —
  the local mount's linearIssueCommentRemotePath rejects UUID-less paths, so
  the old instruction produced files that never became visible comments (codex P1)
- integrations.ts + variants.ts: drop redundant "userId" from the Slack DM
  payload (path-derived; matches the discovery schema) (gemini)
- integrations.ts: remove the stale <channelDir> note — the emitted path is
  already concrete (gemini)
- report.ts: add 'prescriptive' to VARIANT_ORDER so the HTML report column
  sorts last instead of first (gemini)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: apply pr-reviewer fixes for #375

* refactor(integrations): make prescriptiveSpawnInstructions fully adapter-driven

Removes the hardcoded per-provider branches (and the interim curated payload
map) from prescriptiveSpawnInstructions. Writable resources + path templates
now come entirely from each provider's discovery `.adapter.md` (shipped by
relayfile-adapters), and payload shape is pointed at that resource's discovery
`.create.example.json`. No per-provider knowledge lives in pear, so a new
integration works with zero code change here.

- Parse the adapter doc's "Writable resources" section (provider-agnostic)
- Resolve each resource's concrete, in-scope path from the integration's
  writeback mount roots; preserve {id} placeholders for nested resources
- Point at the adapter's create example for fields instead of inlining payloads
- Graceful fallback to a discovery pointer when the adapter doc isn't mounted

Known gap tracked upstream: the local mount currently serves discovery inferred
from synced read records (which omits required write fields like Slack's
`text`), so the pointed-at example is imperfect until that's fixed —
AgentWorkforce/relayfile#299. The adapters already publish correct write-shaped
discovery; the fix belongs in the mount/sync pipeline, not pear.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: remove stray bot-added incident file

agent-relay-code[bot] pushed an unrelated mount-root "incident" note (from its
own Daytona sandbox) onto this PR; it references a non-existent doc and has
nothing to do with the prescriptive-spawn/evals change. Removing it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: agent-relay-code[bot] <agent-relay-code[bot]@users.noreply.github.com>

Author: 3 people

.gitignore

@@ -21,3 +21,6 @@ tests/playwright/screenshots/
 
 # Local factory runtime config (not for commit)
 factory.config.json
+
+evals/reports/*.json
+evals/reports/*.html

evals/fixture.ts

@@ -0,0 +1,55 @@
+/**
+ * Creates a fresh temp directory with a fake .integrations/ mount structure.
+ *
+ * Discovery schemas are copied from evals/fixtures/discovery/ so eval agents
+ * see consistent, realistic schemas without depending on a live mount.
+ * Provider dirs (slack/channels/..., linear/issues/) are empty and writable.
+ */
+
+import { cpSync, existsSync, mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'
+import { dirname, join } from 'node:path'
+import { fileURLToPath } from 'node:url'
+import os from 'node:os'
+
+import { snapshotMount, newMountFiles } from '@agent-relay/evals/scoring/mount'
+
+const __dirname = dirname(fileURLToPath(import.meta.url))
+const BUNDLED_DISCOVERY = join(__dirname, 'fixtures', 'discovery')
+
+// Stable fake IDs used across all scenarios
+export const EVAL_CHANNEL_ID = 'C12345EVAL'
+export const EVAL_CHANNEL_SLUG = 'general'
+export const EVAL_CHANNEL_DIR = `${EVAL_CHANNEL_ID}__${EVAL_CHANNEL_SLUG}`
+export const EVAL_USER_ID = 'U67890EVAL'
+export const EVAL_ISSUE_ID = 'ARC-123EVL'
+
+/**
+ * Create a fresh temp directory with the fake mount.
+ * Returns the absolute path to the temp dir.
+ */
+export function createFixture({ claudeMd }: { claudeMd?: string | null } = {}): string {
+  const tmpDir = mkdtempSync(join(os.tmpdir(), 'pear-eval-'))
+
+  // Copy bundled discovery schemas
+  const discoveryDest = join(tmpDir, '.integrations', 'discovery')
+  mkdirSync(discoveryDest, { recursive: true })
+  if (existsSync(BUNDLED_DISCOVERY)) {
+    cpSync(BUNDLED_DISCOVERY, discoveryDest, { recursive: true })
+  }
+
+  // Writable provider dirs — empty, agent creates files here
+  mkdirSync(join(tmpDir, '.integrations', 'slack', 'channels', EVAL_CHANNEL_DIR, 'messages'), { recursive: true })
+  mkdirSync(join(tmpDir, '.integrations', 'slack', 'users', EVAL_USER_ID, 'messages'), { recursive: true })
+  mkdirSync(join(tmpDir, '.integrations', 'linear', 'issues'), { recursive: true })
+  // Pre-create the comment subpath so s05 has a valid target dir
+  mkdirSync(join(tmpDir, '.integrations', 'linear', 'issues', EVAL_ISSUE_ID, 'comments'), { recursive: true })
+
+  if (claudeMd) {
+    writeFileSync(join(tmpDir, 'CLAUDE.md'), claudeMd)
+  }
+
+  return tmpDir
+}
+
+// Re-export for convenience so callers don't need to import from two places
+export { snapshotMount, newMountFiles }

evals/fixtures/discovery/linear/.adapter.md

@@ -0,0 +1,45 @@
+# linear writeback adapter
+
+This provider supports file-native writeback. Before creating or
+updating a resource, read its schema and example files so you do not
+guess the payload shape.
+
+## Writable resources
+
+### issues — `.integrations/linear/issues`
+
+- schema: `discovery/linear/issues/.schema.json`
+- create example: `discovery/linear/issues/.create.example.json`
+- update example: `discovery/linear/issues/.update.example.json`
+- delete example: `discovery/linear/issues/.delete.example.json`
+
+### comments — `.integrations/linear/issues/{issueId}/comments`
+
+- schema: `discovery/linear/issues/{issueId}/comments/.schema.json`
+- create example: `discovery/linear/issues/{issueId}/comments/.create.example.json`
+
+## Actions
+
+### Create
+Write a new JSON file (any filename, no leading dot) under the resource path.
+Example: `.integrations/linear/issues/new-bug.json`
+
+### Update
+Write a JSON file with `"_action": "update"` and the `"id"` of the record to
+update, plus the fields to change.
+Example: `.integrations/linear/issues/update-ARC-123.json`
+
+### Delete
+Write a JSON file with `"_action": "delete"` and the `"id"` of the record.
+Example: `.integrations/linear/issues/delete-ARC-123.json`
+
+### Comment
+Write a JSON file under the issue's comments path with `"issueId"` and `"body"`.
+Example: `.integrations/linear/issues/ARC-123EVL/comments/comment-1.json`
+
+## Contract
+
+- Read `<resource>/.schema.json` (JSON Schema draft 2020-12). Fields
+  with `readOnly: true` are server-managed; never set them.
+- Read `<resource>/.create.example.json` for a minimal valid create payload.
+- Do NOT write files inside `discovery/` — that directory is read-only schema reference.

evals/fixtures/discovery/linear/issues/.create.example.json

@@ -0,0 +1,5 @@
+{
+  "title": "Issue title",
+  "description": "Describe the issue here",
+  "priority": 2
+}

evals/fixtures/discovery/linear/issues/.delete.example.json

@@ -0,0 +1,4 @@
+{
+  "id": "ISSUE_ID_HERE",
+  "_action": "delete"
+}

evals/fixtures/discovery/linear/issues/.schema.json

@@ -0,0 +1,18 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "linear issues",
+  "description": "Record shape for /linear/issues. Fields marked readOnly are server-managed; omit them from create payloads.",
+  "type": "object",
+  "additionalProperties": true,
+  "properties": {
+    "id": { "type": "string", "readOnly": true },
+    "identifier": { "type": "string", "readOnly": true },
+    "title": { "type": "string" },
+    "description": { "type": "string" },
+    "priority": { "type": "integer" },
+    "assignee_name": { "type": "string" },
+    "labelIds": { "type": "array", "items": { "type": "string" } },
+    "createdAt": { "type": "string", "readOnly": true },
+    "creatorId": { "type": "string", "readOnly": true }
+  }
+}

evals/fixtures/discovery/linear/issues/.update.example.json

@@ -0,0 +1,6 @@
+{
+  "id": "ISSUE_ID_HERE",
+  "_action": "update",
+  "title": "Updated title",
+  "priority": 1
+}

evals/fixtures/discovery/linear/issues/{issueId}/comments/.create.example.json

@@ -0,0 +1,4 @@
+{
+  "issueId": "ISSUE_ID_HERE",
+  "body": "Your comment text here."
+}

evals/fixtures/discovery/linear/issues/{issueId}/comments/.schema.json

@@ -0,0 +1,15 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "linear issue comments",
+  "description": "Record shape for /linear/issues/{issueId}/comments. Fields marked readOnly are server-managed; omit them from create payloads.",
+  "type": "object",
+  "additionalProperties": true,
+  "properties": {
+    "id": { "type": "string", "readOnly": true },
+    "issueId": { "type": "string" },
+    "body": { "type": "string" },
+    "createdAt": { "type": "string", "readOnly": true },
+    "userId": { "type": "string", "readOnly": true }
+  },
+  "required": ["issueId", "body"]
+}

evals/fixtures/discovery/slack/.adapter.md

@@ -0,0 +1,39 @@
+# slack writeback adapter
+
+This provider supports file-native writeback. Before creating or
+updating a resource, read its schema and create example so you do not
+guess the payload shape.
+
+## Writable resources
+
+### messages — `/slack/channels/{channelId}/messages`
+
+- schema: `discovery/slack/channels/{channelId}/messages/.schema.json`
+- create example: `discovery/slack/channels/{channelId}/messages/.create.example.json`
+
+### reactions — `/slack/channels/{channelId}/messages/{messageTs}/reactions`
+
+- schema: `discovery/slack/channels/{channelId}/messages/{messageTs}/reactions/.schema.json`
+- create example: `discovery/slack/channels/{channelId}/messages/{messageTs}/reactions/.create.example.json`
+
+### replies — `/slack/channels/{channelId}/messages/{messageTs}/replies`
+
+- schema: `discovery/slack/channels/{channelId}/messages/{messageTs}/replies/.schema.json`
+- create example: `discovery/slack/channels/{channelId}/messages/{messageTs}/replies/.create.example.json`
+
+### direct-messages — `/slack/users/{userId}/messages`
+
+- schema: `discovery/slack/users/{userId}/messages/.schema.json`
+- create example: `discovery/slack/users/{userId}/messages/.create.example.json`
+
+## Contract
+
+- Read `<resource>/.schema.json` (JSON Schema draft 2020-12). Fields
+  with `readOnly: true` are server-managed; never set them.
+- Read `<resource>/.create.example.json` for a minimal valid create
+  payload (read-only fields already omitted).
+- Schemas are inferred from synced records and refine on each
+  sync; treat unknown extra fields as allowed.
+- Write a new JSON file (any filename) under the target resource path to
+  dispatch the action. Example: `.integrations/slack/channels/C12345__general/messages/my-message.json`
+- Do NOT write files inside `discovery/` — that directory is read-only schema reference.