From ad3d8b672c3a4a341b4ec155bee48df861b52fbb Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Thu, 2 Jul 2026 14:53:15 -0400 Subject: [PATCH] feat(cli): add runtime fallback to agents repo for unconfigured agents When an agent is not registered in config.yaml, resolveAgentSource now attempts to fetch the latest harness from fullsend-ai/agents before falling back to the scaffold-embedded harness on disk. The fallback resolves the main branch HEAD SHA via the forge.Client interface, validates the SHA format, checks the org allowlist before any outbound fetch, and caches the fetched content directly. Supply- chain integrity relies on commit-pinned URLs, TLS transport, and the org allowlist. Only known first-party agents (triage, code, fix, review, retro, prioritize) are eligible for fallback; the fallback is skipped in offline mode or when no forge client is available. This is a transitional mechanism for the agent extraction (docs/plans/agent-extraction-to-agents-repo.md) and will be removed once all users have migrated to config-driven registration (ADR 0058 Phase 5). Signed-off-by: Greg Allen Signed-off-by: Claude Signed-off-by: Greg Allen --- docs/ADRs/0058-agent-registration.md | 1 + docs/architecture.md | 2 +- docs/plans/agent-extraction-to-agents-repo.md | 781 ++++++++++++++++++ docs/plans/agent-registration.md | 12 +- internal/cli/run.go | 149 +++- internal/cli/run_test.go | 331 +++++++- 6 files changed, 1265 insertions(+), 11 deletions(-) create mode 100644 docs/plans/agent-extraction-to-agents-repo.md diff --git a/docs/ADRs/0058-agent-registration.md b/docs/ADRs/0058-agent-registration.md index d7249a776..ee52c0b7c 100644 --- a/docs/ADRs/0058-agent-registration.md +++ b/docs/ADRs/0058-agent-registration.md @@ -78,6 +78,7 @@ phasing, schema details, CLI behavior, and migration mechanics. - The additive merge model allows agents to be extracted from the scaffold one at a time without disrupting existing installations. - Per-repo installs no longer need org config for remote resource validation. - No forced migration — empty config falls back to scaffold discovery until populated. +- **Transitional agents-repo fallback:** During the [agent extraction](../plans/agent-extraction-to-agents-repo.md), a runtime fallback resolves known first-party agents from `fullsend-ai/agents` when not in config. This avoids requiring config changes from existing users during extraction. The fallback will be removed once all users have migrated to config-driven registration (Phase 5 / extraction plan Step 7). - The `agents` YAML key was previously used in `OrgConfig` with a different schema (role/name/slug identity tuples, removed by ADR 0045 Phase 4). The new schema (URL/path source entries) is incompatible; a custom unmarshaler detects and rejects old-format entries with a clear error message. ## References diff --git a/docs/architecture.md b/docs/architecture.md index 26c3d2874..ea525d902 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -240,7 +240,7 @@ Fullsend provides a base set of agent definitions. The adopting organization's * **Decided:** - Config-level agent registration: an `agents` list in both `OrgConfig` and `PerRepoConfig` declares agent harness sources as pinned URLs or local paths, replacing compiled-in agent discovery ([ADR 0058](ADRs/0058-agent-registration.md)). -- Runtime resolution: `fullsend run ` looks up the agent in config and loads the harness directly from the URL or path — no intermediate wrapper files on disk. Role and slug come from the harness content itself. +- Runtime resolution: `fullsend run ` resolves agents in three tiers: (1) config entries from `OrgConfig.Agents` (highest priority), (2) runtime fallback to the `fullsend-ai/agents` repository for known first-party agents not in config, (3) scaffold-embedded harnesses on disk. The agents-repo fallback is a transitional mechanism for the [agent extraction](plans/agent-extraction-to-agents-repo.md); it will be removed once all users have migrated to config-driven registration (ADR 0058 Phase 5). - Additive merge: config entries overlay scaffold-discovered agents (config wins on name collision), enabling gradual extraction of first-party agents without disrupting existing installations. Builds on [ADR 0045](ADRs/0045-forge-portable-harness-schema.md) harness identity model. - CLI management: `fullsend agent add/list/update/remove` manages config entries and auto-pins URLs to a commit SHA with an integrity hash. diff --git a/docs/plans/agent-extraction-to-agents-repo.md b/docs/plans/agent-extraction-to-agents-repo.md new file mode 100644 index 000000000..609cc88cd --- /dev/null +++ b/docs/plans/agent-extraction-to-agents-repo.md @@ -0,0 +1,781 @@ +# Plan: Extract remaining agents from fullsend to fullsend-ai/agents + +**Date:** 2026-07-02 +**Implements:** [ADR 0058](../ADRs/0058-agent-registration.md) Phase 4 (extended to all agents) +**Prerequisite:** ADR 0058 Phases 1–3 are already implemented. + +## Current state + +### Triage agent (already extracted) + +The triage agent has been replicated to `fullsend-ai/agents` and is +running from that repo. The scaffold still contains the triage source +files (Phase 4 deletion has not been done yet). + +### Triage agent: fullsend vs. agents repo differences + +Files that are **identical** between `upstream/main` scaffold and +`fullsend-ai/agents@main`: + +| File | Path in scaffold | Path in agents repo | +|------|-----------------|-------------------| +| Agent prompt | `agents/triage.md` | `agents/triage.md` | +| Pre-script | `scripts/pre-triage.sh` | `scripts/pre-triage.sh` | +| Post-script | `scripts/post-triage.sh` | `scripts/post-triage.sh` | +| Post-script tests | `scripts/post-triage-test.sh` | `scripts/post-triage-test.sh` | +| Output schema | `schemas/triage-result.schema.json` | `schemas/triage-result.schema.json` | +| Validation script | `scripts/validate-output-schema.sh` | `scripts/validate-output-schema.sh` | +| Validation tests | `scripts/validate-output-schema-test.sh` | `scripts/validate-output-schema-test.sh` | +| Issue-labels skill | `skills/issue-labels/SKILL.md` | `skills/issue-labels/SKILL.md` | + +Files that **differ** between the two repos: + +#### 1. `harness/triage.yaml` — three changes required for external-repo operation + +```diff + host_files: +- - src: env/gcp-vertex.env ++ - src: common/env/gcp-vertex.env + dest: /sandbox/workspace/.env.d/gcp-vertex.env + expand: true + - src: ${GOOGLE_APPLICATION_CREDENTIALS} + dest: /tmp/.gcp-credentials.json + - src: ${GCP_OIDC_TOKEN_FILE} + dest: /sandbox/workspace/.gcp-oidc-token + optional: true ++ - src: env/triage.env ++ dest: /sandbox/workspace/.env.d/triage.env ++ expand: true + + validation_loop: + script: scripts/validate-output-schema.sh ++ schema: schemas/triage-result.schema.json + max_iterations: 2 + +-env: +- runner: +- FULLSEND_OUTPUT_SCHEMA: ${FULLSEND_DIR}/schemas/triage-result.schema.json +``` + +**Why each change is needed:** + +1. **`env/gcp-vertex.env` → `common/env/gcp-vertex.env`**: In the + scaffold, each agent references `env/gcp-vertex.env` directly. In + the agents repo, shared env files are placed in `common/env/` to + avoid duplication when multiple agents coexist. The content is + identical. + +2. **Added `env/triage.env` host_file**: In the scaffold, the forge + section's `env.runner`/`env.sandbox` injects `GITHUB_ISSUE_URL` + and `GH_TOKEN` into the agent's environment. In the agents repo, + these are instead provided via a dedicated env file + (`env/triage.env`) loaded as a host_file. This is the + self-contained alternative to relying on the forge section's env + injection, which requires `${FULLSEND_DIR}` resolution that only + works when the harness is loaded from the scaffold. + +3. **`validation_loop.schema` replaces `env.runner.FULLSEND_OUTPUT_SCHEMA`**: + The scaffold sets `FULLSEND_OUTPUT_SCHEMA` as a runner env var + using `${FULLSEND_DIR}` expansion. In an external repo, + `${FULLSEND_DIR}` does not resolve to the correct path. The + `validation_loop.schema` field tells the harness to set + `FULLSEND_OUTPUT_SCHEMA` itself after resolving the schema path + relative to the harness file — a feature that was added to + `run.go:validationEnv()` to support exactly this use case. + +#### 2. `policies/triage.yaml` — cosmetic only + +The scaffold version has a YAML document marker (`---`) at line 1; +the agents repo omits it. No functional difference. When extracting +remaining agents, include the `---` document marker to stay +consistent with the scaffold originals. + +#### 3. Additional files in agents repo (not in scaffold) + +| File | Purpose | +|------|---------| +| `common/env/gcp-vertex.env` | Shared GCP Vertex env template (same content as scaffold's `env/gcp-vertex.env`) | +| `env/triage.env` | Agent-specific env file exporting `GITHUB_ISSUE_URL` and `GH_TOKEN` | +| `.github/workflows/fullsend.yaml` | Fullsend shim workflow (auto-managed by enrollment) | +| `docs/triage.md` | Agent documentation (slightly simplified from scaffold's `docs/agents/triage.md`) | +| `docs/icons/triage.png` | Agent icon | + +#### 4. Files NOT present in agents repo (remain in fullsend) + +| File | Why it stays | +|------|-------------| +| `env/gcp-vertex.env` | Replaced by `common/env/gcp-vertex.env` in agents repo | +| `.github/workflows/triage.yml` | Scaffold workflow template with placeholders; not needed in agents repo | +| `.github/workflows/reusable-triage.yml` | Reusable workflow stays in fullsend; agents repo does not host workflow infrastructure | +| `eval/triage/` | Eval framework stays in fullsend for now | + +--- + +## Agents remaining to extract + +| Agent | Skills | Plugins | Image | Env files | Shared skills | +|-------|--------|---------|-------|-----------|---------------| +| **code** | `code-implementation` | `gopls-lsp` | `fullsend-code:latest` | `code-agent.env`, `gcp-vertex.env` | — | +| **fix** | `fix-review` | — | `fullsend-code:latest` | `fix-agent.env`, `gcp-vertex.env` | — | +| **review** | `pr-review`, `code-review`, `docs-review`, `issue-labels` | — | `fullsend-code:latest` | `review.env`, `gcp-vertex.env` | `issue-labels` (shared with triage) | +| **retro** | `retro-analysis`, `finding-agent-runs`, `agent-scaffolding`, `autonomy-readiness` | — | `fullsend-sandbox:latest` | `retro.env`, `gcp-vertex.env` | — | +| **prioritize** | — | — | `fullsend-sandbox:latest` | `gcp-vertex.env` | — | + +### Skills inventory and sharing + +``` +Skill Used by Files +───────────────────── ────────── ───────────────────────────────────────── +issue-labels triage,review skills/issue-labels/SKILL.md +code-implementation code skills/code-implementation/SKILL.md +code-review review skills/code-review/SKILL.md +docs-review review skills/docs-review/SKILL.md +fix-review fix skills/fix-review/SKILL.md +pr-review review skills/pr-review/SKILL.md + skills/pr-review/meta-prompt.md + skills/pr-review/sub-agents/*.md (7 files) +retro-analysis retro skills/retro-analysis/SKILL.md +finding-agent-runs retro skills/finding-agent-runs/SKILL.md +agent-scaffolding retro skills/agent-scaffolding/SKILL.md +autonomy-readiness retro skills/autonomy-readiness/SKILL.md +``` + +### Plugin inventory + +``` +Plugin Used by Files +───────────────────── ────────── ───────────────────────── +gopls-lsp code plugins/gopls-lsp/.lsp.json + plugins/gopls-lsp/plugin.json +``` + +--- + +## Preserving git history during extraction + +Files must be moved with their commit history intact so that +`git log --follow` and `git blame` work in the agents repo. A plain +file copy loses all history. Use `git filter-repo` to extract the +relevant paths from fullsend into a temporary clone, rewrite paths +to match the agents repo layout, and merge the result. + +### Procedure (run once per agent or batched for all) + +```bash +# 1. Create a disposable clone of fullsend for history extraction. +git clone --no-checkout https://github.com/fullsend-ai/fullsend.git /tmp/fullsend-extract +cd /tmp/fullsend-extract + +# 2. Use git filter-repo to keep only the agent's files and rewrite +# paths from the scaffold prefix to the agents repo root. +# +# --path selects files to keep (repeat for each path). +# --path-rename strips the scaffold prefix so the files land at +# the correct location in the agents repo. +# +# Example for the code agent: +git filter-repo \ + --path internal/scaffold/fullsend-repo/agents/code.md \ + --path internal/scaffold/fullsend-repo/harness/code.yaml \ + --path internal/scaffold/fullsend-repo/policies/code.yaml \ + --path internal/scaffold/fullsend-repo/schemas/code-result.schema.json \ + --path internal/scaffold/fullsend-repo/scripts/pre-code.sh \ + --path internal/scaffold/fullsend-repo/scripts/post-code.sh \ + --path internal/scaffold/fullsend-repo/scripts/post-code-test.sh \ + --path internal/scaffold/fullsend-repo/env/code-agent.env \ + --path internal/scaffold/fullsend-repo/skills/code-implementation/ \ + --path internal/scaffold/fullsend-repo/plugins/gopls-lsp/ \ + --path-rename internal/scaffold/fullsend-repo/: \ + --force + +# Also include docs from the top-level docs/ directory: +# (run a second filter-repo pass or handle docs separately) + +# 3. From the agents repo, add the filtered clone as a remote and +# merge its history. +cd /path/to/fullsend-agents +git remote add extract /tmp/fullsend-extract +git fetch extract +git merge extract/main --allow-unrelated-histories \ + -m "feat: import code agent with history from fullsend-ai/fullsend" +git remote remove extract + +# 4. Apply harness adaptations (see "Harness adaptation pattern") as +# a follow-up commit on the same branch. + +# 5. Clean up. +rm -rf /tmp/fullsend-extract +``` + +### Path renames + +`--path-rename internal/scaffold/fullsend-repo/:` strips the scaffold +prefix, placing files at the repo root. Files that need further +renaming (e.g., `env/code-agent.env` → `env/code.env`) should use an +additional `--path-rename` flag: + +```bash +--path-rename env/code-agent.env:env/code.env +``` + +### Docs and icons + +Agent docs live at `docs/agents/.md` in fullsend (outside the +scaffold prefix). Include them with a separate `--path` and rename: + +```bash +--path docs/agents/code.md \ +--path docs/agents/icons/coder.png \ +--path-rename docs/agents/:docs/ +``` + +### Batching multiple agents + +To extract all remaining agents in a single pass, list all paths +for all agents in one `git filter-repo` invocation. This produces a +single merge commit with combined history. Individual harness +adaptations can then be done as separate commits. + +### Shared files (gcp-vertex.env, validate-output-schema.sh) + +Files already present in the agents repo (from the triage extraction) +should NOT be re-imported — they would create merge conflicts. Exclude +them from the `--path` list: + +- `env/gcp-vertex.env` — already at `common/env/gcp-vertex.env` +- `scripts/validate-output-schema.sh` — already present +- `scripts/validate-output-schema-test.sh` — already present +- `skills/issue-labels/` — already present + +If the scaffold version has diverged from the agents repo copy, +reconcile manually after the merge. + +### Verifying history + +After the merge, verify history is intact: + +```bash +git log --follow -- agents/code.md +git log --follow -- scripts/post-code.sh +``` + +Each file should show commits from its life in the fullsend repo +(under the scaffold prefix) followed by the merge commit. + +--- + +## Per-agent file manifest + +For each agent, these files must be extracted to the agents repo with +history preserved (see above). The table shows the scaffold source +path (relative to `internal/scaffold/fullsend-repo/`) and the target +path in the agents repo. + +### Code agent + +| Scaffold path | Agents repo path | Notes | +|---------------|-----------------|-------| +| `agents/code.md` | `agents/code.md` | Copy as-is | +| `harness/code.yaml` | `harness/code.yaml` | Needs adaptation (see below) | +| `policies/code.yaml` | `policies/code.yaml` | Copy as-is (retain `---` document marker) | +| `schemas/code-result.schema.json` | `schemas/code-result.schema.json` | Copy as-is | +| `scripts/pre-code.sh` | `scripts/pre-code.sh` | Copy as-is | +| `scripts/post-code.sh` | `scripts/post-code.sh` | Copy as-is | +| `scripts/post-code-test.sh` | `scripts/post-code-test.sh` | Copy as-is | +| `env/code-agent.env` | `env/code.env` | Rename for consistency | +| `skills/code-implementation/` | `skills/code-implementation/` | Copy directory | +| `plugins/gopls-lsp/` | `plugins/gopls-lsp/` | Copy directory | +| `docs/agents/code.md` | `docs/code.md` | Adapt paths/links | +| `docs/agents/icons/coder.png` | `docs/icons/coder.png` | Copy | + +### Fix agent + +| Scaffold path | Agents repo path | Notes | +|---------------|-----------------|-------| +| `agents/fix.md` | `agents/fix.md` | Copy as-is | +| `harness/fix.yaml` | `harness/fix.yaml` | Needs adaptation | +| `policies/fix.yaml` | `policies/fix.yaml` | Copy as-is | +| `schemas/fix-result.schema.json` | `schemas/fix-result.schema.json` | Copy as-is | +| `scripts/pre-fix.sh` | `scripts/pre-fix.sh` | Copy as-is | +| `scripts/post-fix.sh` | `scripts/post-fix.sh` | Copy as-is | +| `scripts/post-fix-test.sh` | `scripts/post-fix-test.sh` | Copy as-is | +| `env/fix-agent.env` | `env/fix.env` | Rename for consistency | +| `skills/fix-review/` | `skills/fix-review/` | Copy directory | +| `docs/agents/fix.md` | `docs/fix.md` | Adapt paths/links | + +### Review agent + +| Scaffold path | Agents repo path | Notes | +|---------------|-----------------|-------| +| `agents/review.md` | `agents/review.md` | Copy as-is | +| `harness/review.yaml` | `harness/review.yaml` | Needs adaptation | +| `policies/review.yaml` | `policies/review.yaml` | Copy as-is | +| `schemas/review-result.schema.json` | `schemas/review-result.schema.json` | Copy as-is | +| `scripts/pre-review.sh` | `scripts/pre-review.sh` | Copy as-is | +| `scripts/post-review.sh` | `scripts/post-review.sh` | Copy as-is | +| `scripts/post-review-test.sh` | `scripts/post-review-test.sh` | Copy as-is | +| `env/review.env` | `env/review.env` | Copy as-is | +| `skills/pr-review/` | `skills/pr-review/` | Copy full directory (includes sub-agents) | +| `skills/code-review/` | `skills/code-review/` | Copy directory | +| `skills/docs-review/` | `skills/docs-review/` | Copy directory | +| `skills/issue-labels/` | Already exists | Shared with triage — already in agents repo | +| `docs/agents/review.md` | `docs/review.md` | Adapt paths/links | +| `docs/agents/icons/review.png` | `docs/icons/review.png` | Copy | + +### Retro agent + +| Scaffold path | Agents repo path | Notes | +|---------------|-----------------|-------| +| `agents/retro.md` | `agents/retro.md` | Copy as-is | +| `harness/retro.yaml` | `harness/retro.yaml` | Needs adaptation | +| `policies/retro.yaml` | `policies/retro.yaml` | Copy as-is | +| `schemas/retro-result.schema.json` | `schemas/retro-result.schema.json` | Copy as-is | +| `scripts/pre-retro.sh` | `scripts/pre-retro.sh` | Copy as-is | +| `scripts/post-retro.sh` | `scripts/post-retro.sh` | Copy as-is | +| `scripts/post-retro-test.sh` | `scripts/post-retro-test.sh` | Copy as-is | +| `env/retro.env` | `env/retro.env` | Copy as-is | +| `skills/retro-analysis/` | `skills/retro-analysis/` | Copy directory | +| `skills/finding-agent-runs/` | `skills/finding-agent-runs/` | Copy directory | +| `skills/agent-scaffolding/` | `skills/agent-scaffolding/` | Copy directory | +| `skills/autonomy-readiness/` | `skills/autonomy-readiness/` | Copy directory | +| `docs/agents/retro.md` | `docs/retro.md` | Adapt paths/links | +| `docs/agents/icons/retro.png` | `docs/icons/retro.png` | Copy | + +### Prioritize agent + +| Scaffold path | Agents repo path | Notes | +|---------------|-----------------|-------| +| `agents/prioritize.md` | `agents/prioritize.md` | Copy as-is | +| `harness/prioritize.yaml` | `harness/prioritize.yaml` | Needs adaptation | +| `policies/prioritize.yaml` | `policies/prioritize.yaml` | Copy as-is | +| `schemas/prioritize-result.schema.json` | `schemas/prioritize-result.schema.json` | Copy as-is | +| `scripts/pre-prioritize.sh` | `scripts/pre-prioritize.sh` | Copy as-is | +| `scripts/post-prioritize.sh` | `scripts/post-prioritize.sh` | Copy as-is | +| `scripts/post-prioritize-test.sh` | `scripts/post-prioritize-test.sh` | Copy as-is | +| `docs/agents/prioritize.md` | `docs/prioritize.md` | Adapt paths/links | +| `docs/agents/icons/prioritize.png` | `docs/icons/prioritize.png` | Copy | + +--- + +## Harness adaptation pattern + +Every harness YAML needs three categories of changes when moved to +the agents repo. These are the same changes already applied to the +triage agent's harness: + +### 1. Move `env/gcp-vertex.env` reference to `common/env/` + +```yaml +# Scaffold (before): +host_files: + - src: env/gcp-vertex.env + dest: /sandbox/workspace/.env.d/gcp-vertex.env + expand: true + +# Agents repo (after): +host_files: + - src: common/env/gcp-vertex.env + dest: /sandbox/workspace/.env.d/gcp-vertex.env + expand: true +``` + +`common/env/gcp-vertex.env` already exists in the agents repo. All +agents share this file. + +### 2. Add agent-specific env file as host_file + +Each agent that has forge-section env vars (`GITHUB_ISSUE_URL`, +`GH_TOKEN`, `TARGET_BRANCH`, etc.) needs a dedicated env file that +exports these variables. This replaces the scaffold's +`forge.github.env.sandbox` mechanism. + +Create `env/.env` with the variables the agent needs: + +**Example — `env/code.env`:** +```bash +export GITHUB_ISSUE_URL="${GITHUB_ISSUE_URL}" +export GH_TOKEN=${GH_TOKEN} +``` + +Then add it to `host_files`: +```yaml +host_files: + - src: env/code.env + dest: /sandbox/workspace/.env.d/code.env + expand: true +``` + +The specific variables per agent are documented in each agent's +`forge.github.env` section in the scaffold harness. + +### 3. Use `validation_loop.schema` instead of `env.runner.FULLSEND_OUTPUT_SCHEMA` + +```yaml +# Scaffold (before): +validation_loop: + script: scripts/validate-output-schema.sh + max_iterations: 2 + +env: + runner: + FULLSEND_OUTPUT_SCHEMA: ${FULLSEND_DIR}/schemas/-result.schema.json + +# Agents repo (after): +validation_loop: + script: scripts/validate-output-schema.sh + schema: schemas/-result.schema.json + max_iterations: 2 +``` + +The `validation_loop.schema` field tells the harness to resolve the +schema path relative to the harness file and set +`FULLSEND_OUTPUT_SCHEMA` automatically. This avoids the +`${FULLSEND_DIR}` reference that doesn't resolve correctly for +externally-loaded harnesses. + +### 4. Handle remaining `env.runner` / `runner_env` variables + +Some agents have additional runner env vars beyond +`FULLSEND_OUTPUT_SCHEMA`. These need to be evaluated case-by-case: + +- **Code agent**: `CODE_ALLOWED_TARGET_BRANCHES` — keep in + `env.runner` (or move to `env/code.env` if appropriate) +- **Fix agent**: `TARGET_BRANCH`, `TRIGGER_SOURCE`, + `HUMAN_INSTRUCTION`, `FIX_ITERATION`, `REVIEW_BODY_FILE`, + `PRE_AGENT_HEAD` — these are set by the reusable workflow and + passed through `env.runner`; keep in harness `env.runner` +- **Retro/Prioritize**: check for any runner env vars + +### 5. Retain `forge` section + +The `forge.github` section should be preserved in the harness. It +contains forge-specific pre/post script paths and env var mappings +that the harness runtime uses when running on GitHub. The forge +section env vars are the authoritative source for what gets passed +to pre/post scripts on the runner side. + +--- + +## Shared resources in the agents repo + +### `common/` directory + +Shared files that multiple agents reference go in `common/`. Currently: +- `common/env/gcp-vertex.env` — GCP Vertex AI configuration + +### `scripts/validate-output-schema.sh` + +This script is shared by all agents. It already exists in the agents +repo. Each new agent reuses it without duplication. + +### `scripts/validate-output-schema-test.sh` + +This test file validates the schema validation script against multiple +agent schemas. It already references `fix-result.schema.json` and +`review-result.schema.json` — those schemas must exist in the agents +repo for the tests to work. As agents are added, the tests will +naturally cover more schemas. + +### `skills/issue-labels/` + +Shared between triage and review. Already exists in agents repo. + +--- + +## Execution plan + +### Step 1: Extract code agent (PR in agents repo) + +1. Run `git filter-repo` to extract code agent files with history + (see "Preserving git history during extraction") +2. Merge the extracted history into the agents repo +3. Create `env/code.env` with the agent's required env vars +4. Adapt `harness/code.yaml` per the harness adaptation pattern +5. Adapt `docs/code.md` links +6. Verify history: `git log --follow -- agents/code.md` +7. Run `scripts/post-code-test.sh` to verify post-script works +8. Verify `scripts/validate-output-schema-test.sh` still passes with + the code schema present + +### Step 2: Extract fix agent (PR in agents repo) + +1. Run `git filter-repo` to extract fix agent files with history +2. Merge the extracted history into the agents repo +3. Create `env/fix.env` with required env vars +4. Adapt `harness/fix.yaml` +5. Adapt `docs/fix.md` links +6. Verify history: `git log --follow -- agents/fix.md` +7. Run post-fix tests + +### Step 3: Extract review agent (PR in agents repo) + +1. Run `git filter-repo` to extract review agent files with history + (include `skills/pr-review/`, `skills/code-review/`, + `skills/docs-review/`; exclude `skills/issue-labels/` — already + present) +2. Merge the extracted history into the agents repo +3. Create `env/review.env` with required env vars (may already exist) +4. Adapt `harness/review.yaml` +5. Adapt `docs/review.md` links +6. Verify history: `git log --follow -- agents/review.md` +7. Run post-review tests + +### Step 4: Extract retro agent (PR in agents repo) + +1. Run `git filter-repo` to extract retro agent files with history + (include `skills/retro-analysis/`, `skills/finding-agent-runs/`, + `skills/agent-scaffolding/`, `skills/autonomy-readiness/`) +2. Merge the extracted history into the agents repo +3. Create `env/retro.env` (may already exist in scaffold) +4. Adapt `harness/retro.yaml` +5. Adapt `docs/retro.md` links +6. Verify history: `git log --follow -- agents/retro.md` +7. Run post-retro tests + +### Step 5: Extract prioritize agent (PR in agents repo) + +1. Run `git filter-repo` to extract prioritize agent files with + history +2. Merge the extracted history into the agents repo +3. Adapt `harness/prioritize.yaml` +4. Adapt `docs/prioritize.md` links +5. Verify history: `git log --follow -- agents/prioritize.md` +6. Run post-prioritize tests + +### Step 6: Register agents in fullsend-ai org config (PR in fullsend-ai/.fullsend) + +Convert the fullsend-ai organization to use agents from the agents +repo, mirroring what was already done for the triage agent. + +The current `fullsend-ai/.fullsend/config.yaml` has: + +```yaml +agents: + - source: https://raw.githubusercontent.com/fullsend-ai/agents//harness/triage.yaml#sha256= +allowed_remote_resources: + - https://raw.githubusercontent.com/fullsend-ai/fullsend/ + - https://raw.githubusercontent.com/fullsend-ai/agents/ +``` + +For each extracted agent, register it the same way triage was +registered — using `fullsend agent add` from the `.fullsend` config +repo checkout, or by manually adding a pinned URL entry to the +`agents:` list. + +#### 6a. Add each agent to the config + +Run `fullsend agent add` for each agent (from a checkout of +`fullsend-ai/.fullsend`): + +```bash +fullsend agent add https://github.com/fullsend-ai/agents/blob/main/harness/code.yaml +fullsend agent add https://github.com/fullsend-ai/agents/blob/main/harness/fix.yaml +fullsend agent add https://github.com/fullsend-ai/agents/blob/main/harness/review.yaml +fullsend agent add https://github.com/fullsend-ai/agents/blob/main/harness/retro.yaml +fullsend agent add https://github.com/fullsend-ai/agents/blob/main/harness/prioritize.yaml +``` + +Each `agent add` auto-pins the URL to the current commit SHA of +`fullsend-ai/agents` and computes the `#sha256=` integrity hash. The +`allowed_remote_resources` entry for `fullsend-ai/agents/` already +exists, so no allowlist changes are needed. + +The resulting `config.yaml` `agents:` section should look like: + +```yaml +agents: + - source: https://raw.githubusercontent.com/fullsend-ai/agents//harness/triage.yaml#sha256= + - source: https://raw.githubusercontent.com/fullsend-ai/agents//harness/code.yaml#sha256= + - source: https://raw.githubusercontent.com/fullsend-ai/agents//harness/fix.yaml#sha256= + - source: https://raw.githubusercontent.com/fullsend-ai/agents//harness/review.yaml#sha256= + - source: https://raw.githubusercontent.com/fullsend-ai/agents//harness/retro.yaml#sha256= + - source: https://raw.githubusercontent.com/fullsend-ai/agents//harness/prioritize.yaml#sha256= +``` + +#### 6b. Verify agent resolution + +Run `fullsend agent list` from the `.fullsend` checkout to verify all +agents resolve correctly and show source as the agents repo URL +instead of scaffold: + +```bash +fullsend agent list --fullsend-dir . +``` + +Expected output: all 6 agents listed with their agents-repo URLs, +overriding the scaffold defaults. + +#### 6c. Verify dispatch routing + +Confirm that the `.fullsend` config repo's dispatch workflow and +per-stage workflow files (`triage.yml`, `code.yml`, `fix.yml`, +`review.yml`, `retro.yml`, `prioritize.yml`) do not need changes. +The dispatch routes by `# fullsend-stage:` markers in workflow files +and the reusable workflows resolve agents from config at runtime — +the harness source URL is transparent to dispatch. + +#### 6d. Smoke test each agent + +After the config PR merges, trigger each agent on a test issue/PR in +an enrolled repo (e.g., `fullsend-ai/agents` itself or +`fullsend-ai/experiments`) to verify end-to-end operation: + +| Agent | Trigger | What to verify | +|-------|---------|---------------| +| triage | Open a test issue or `/fs-triage` | Agent runs, posts triage comment, applies labels | +| code | Apply `ready-to-code` label or `/fs-code` | Agent runs, creates branch and commits | +| review | Open a PR or `/fs-review` | Agent runs, posts review | +| fix | Bot review with changes_requested or `/fs-fix` | Agent runs, pushes fix commits | +| retro | Close/merge a PR | Agent runs, posts retro analysis | +| prioritize | `/fs-prioritize` | Agent runs, posts priority scores | + +#### 6e. Roll out to enrolled repos + +Since the `agents:` config is in the org-level `.fullsend` repo, +all enrolled repos under `fullsend-ai` (fullsend, agents, +experiments, metrics) automatically use the new agent sources. No +per-repo changes are needed. + +Verify that each enrolled repo's agent runs succeed by monitoring +the first few natural triggers after the config change merges. Check +the GitHub Actions runs in `fullsend-ai/.fullsend` for any failures. + +#### 6f. Pin management + +After initial rollout, agent URLs are pinned to the commit SHA at +registration time. When the agents repo is updated (bug fixes, +prompt improvements), run `fullsend agent update ` to re-pin +to the latest commit: + +```bash +fullsend agent update triage +fullsend agent update code +# ... etc +``` + +This can be automated via a scheduled workflow or done manually after +verifying changes in the agents repo. + +### Step 7: Transition to authoritative config and remove scaffold agents (ADR 0058 Phase 5) + +**Prerequisite:** All fullsend customers must be upgraded to a +version that supports config-driven agents and have registered +agents-repo agents in their `.fullsend/config.yaml`. Until then, +the scaffold-embedded agents serve as the default fallback — deleting +them would break customers who haven't migrated. + +Once all customers are migrated: + +1. Update install seeding to use agents repo URLs exclusively +2. Remove scaffold fallback from `MergedAgents()` +3. Remove `HarnessNames()` (or restrict to install-time seeding) +4. Remove `HarnessWrappersLayer` +5. Delete scaffold agent files from `internal/scaffold/fullsend-repo/`: + - `agents/*.md` (all 6) + - `harness/*.yaml` (all 6) + - `policies/*.yaml` (all 6) + - `schemas/*-result.schema.json` (all 6) + - `scripts/pre-*.sh`, `scripts/post-*.sh`, `scripts/post-*-test.sh` + (all agent-specific scripts) + - `env/code-agent.env`, `env/fix-agent.env`, `env/review.env`, + `env/retro.env` (agent-specific env files) + - `env/gcp-vertex.env` (moved to agents repo's `common/`) + - All skill directories under `skills/` (moved to agents repo) + - `plugins/gopls-lsp/` (moved to agents repo) +6. Update `internal/scaffold/scaffold.go` — remove deleted files from + `executableFiles` map +7. Update `Makefile` — remove deleted test scripts from `script-test` + target +8. Update `scripts/validate-output-schema-test.sh` — it references + schemas that no longer exist; either update references or move the + test file entirely to the agents repo +9. Update `docs/agents/*.md` — update source links to point to + `fullsend-ai/agents` +10. Update test files: + - `internal/scaffold/scaffold_test.go` + - `internal/scaffold/baseurl_test.go` + - `internal/harness/scaffold_integration_test.go` + - `internal/layers/harnesswrappers_test.go` + - `internal/scaffold/vendormanifest_test.go` + - `internal/layers/workflows_test.go` +11. File tracking issue per the ADR plan + +--- + +## Agent-specific env var inventory + +Variables each agent needs in its env file (derived from +`forge.github.env.sandbox` in each scaffold harness): + +| Agent | Runner vars | Sandbox vars | +|-------|-------------|-------------| +| **triage** | `GITHUB_ISSUE_URL`, `GH_TOKEN` | `GITHUB_ISSUE_URL`, `GH_TOKEN` | +| **code** | `GITHUB_ISSUE_URL`, `GH_TOKEN`, `CODE_ALLOWED_TARGET_BRANCHES` | `GITHUB_ISSUE_URL`, `GH_TOKEN` | +| **fix** | `TARGET_BRANCH`, `TRIGGER_SOURCE`, `HUMAN_INSTRUCTION`, `FIX_ITERATION`, `REVIEW_BODY_FILE`, `PRE_AGENT_HEAD` | `GITHUB_PR_URL`, `GH_TOKEN`, `TARGET_BRANCH` | +| **review** | `GITHUB_PR_URL`, `GH_TOKEN`, `ORG` | `GITHUB_PR_URL`, `GH_TOKEN` | +| **retro** | `GITHUB_PR_URL`, `GH_TOKEN` | `GITHUB_PR_URL`, `GH_TOKEN` | +| **prioritize** | `GITHUB_ISSUE_URL`, `GH_TOKEN`, `ORG`, `PROJECT_NUMBER` | `GITHUB_ISSUE_URL`, `GH_TOKEN` | + +The `env/.env` file in the agents repo should export the +**sandbox** variables (those are what the agent process sees). The +**runner** variables remain in the harness `forge.github.env.runner` +section (they're used by pre/post scripts on the host, not by the +agent itself). + +--- + +## Ordering recommendation + +Extract agents in this order, from simplest to most complex: + +1. **Prioritize** — simplest: no skills, no plugins, sandbox image +2. **Retro** — read-only agent, sandbox image, unique skills +3. **Code** — has plugins (gopls-lsp), code image +4. **Fix** — shares role/slug with code, complex env vars +5. **Review** — most complex: 4 skills (one shared), 7 sub-agents + +Each extraction can be done as an independent PR in the agents repo. +Steps 1–5 are parallelizable (no dependencies between agents), but +doing them in complexity order reduces risk. + +--- + +## Risks and considerations + +1. **Shared skill divergence**: `issue-labels` is used by both triage + and review. It already exists in the agents repo. When review is + extracted, verify it matches the latest scaffold version. If the + scaffold version is updated after extraction, both repos need + syncing until Phase 5 removes the scaffold copies. + +2. **Reusable workflows stay in fullsend**: The `reusable-*.yml` + workflows in `.github/workflows/` remain in the fullsend repo. + They reference agent harnesses at runtime via config, not via + scaffold embedding. This is the design from ADR 0058. + +3. **Eval framework stays in fullsend**: The `eval/` directory with + functional test cases stays in the fullsend repo. Evals test the + harness + agent integration and are run from fullsend CI. + +4. **Scaffold workflow templates stay in fullsend**: The per-stage + workflow stubs (`.github/workflows/triage.yml` etc.) in the + scaffold are templates with `__REUSABLE_WORKFLOW__` and + `__GH_RUNNER__` placeholders. They are instantiated during + enrollment. These stay in fullsend and do not need to be in the + agents repo. + +5. **`validate-output-schema-test.sh` references multiple schemas**: + The test file currently references `fix-result.schema.json` and + `review-result.schema.json` via relative paths. All agent schemas + need to be present in the agents repo for these tests to pass. + This is naturally resolved as agents are extracted. + +6. **AGENTS.md in scaffold**: The scaffold's `AGENTS.md` file + contains shared agent rules that all agents reference. This file + should be copied to the agents repo or made available via the + harness layering mechanism. + +7. **`validate-output-schema.sh` and `validate-output-schema-test.sh` + are shared**: These scripts are agent-agnostic. They already exist + in the agents repo. As more schemas are added, the test file + exercises more of them. diff --git a/docs/plans/agent-registration.md b/docs/plans/agent-registration.md index 9c676f81f..2356dc8d0 100644 --- a/docs/plans/agent-registration.md +++ b/docs/plans/agent-registration.md @@ -250,7 +250,17 @@ When `fullsend run ` is invoked: Role and slug come from the harness content itself. 4. **Local path source:** resolve the path relative to `.fullsend/` and load the harness file directly. -5. **Scaffold source** (no config override): load the scaffold harness +5. **Agents repo fallback** (transitional): for known first-party + agents not found in config, resolve the latest harness from + `fullsend-ai/agents` at runtime. The fallback resolves the `main` + branch HEAD SHA via the forge client, constructs a commit-pinned + URL, checks the org allowlist, fetches via `fetch.FetchURL`, and + caches content directly. Supply-chain integrity relies on the + commit-pinned URL, TLS transport, and the org allowlist. This tier + exists to support the [agent extraction](agent-extraction-to-agents-repo.md) + without requiring config changes from existing users and will be + removed once all users have migrated (Step 7). +6. **Scaffold source** (no config override): load the scaffold harness as today. The `--harness` flag, when given a name instead of a file path, uses diff --git a/internal/cli/run.go b/internal/cli/run.go index ca4f5a591..d3d1b25af 100644 --- a/internal/cli/run.go +++ b/internal/cli/run.go @@ -49,8 +49,39 @@ const ( // metricsFile is the filename written to the run directory with behavioral metrics. metricsFile = "metrics.json" + + // Default agents repository for runtime fallback when an agent is not + // registered in config. The binary resolves the latest commit SHA from + // this repo and fetches the harness dynamically. + defaultAgentsRepoOwner = "fullsend-ai" + defaultAgentsRepoName = "agents" + defaultAgentsRepoBranch = "main" ) +// defaultAgentsRepoURLPrefix is the base URL for fetching agent harnesses +// from the agents repository. It is a var (not const) to allow test overrides. +var defaultAgentsRepoURLPrefix = "https://raw.githubusercontent.com/fullsend-ai/agents/" + +// validHexSHA matches a 40-character lowercase hex SHA (git commit hash). +var validHexSHA = regexp.MustCompile(`^[0-9a-f]{40}$`) + +// defaultAgentsRepoKnownAgents lists first-party agents available in the +// fullsend-ai/agents repository. Only these agents are eligible for the +// runtime fallback — custom agents are never tried against the agents repo. +// +// This is a transitional mechanism to support agent extraction (see +// docs/plans/agent-extraction-to-agents-repo.md). It will be removed once +// all users have migrated to config-driven agent registration (ADR 0058 +// Phase 5 / extraction plan Step 7). +var defaultAgentsRepoKnownAgents = map[string]bool{ + "triage": true, + "code": true, + "fix": true, + "review": true, + "retro": true, + "prioritize": true, +} + // statusMintToken is the test seam for minting tokens. Shared by both // setupStatusNotifier (status comment tokens) and mintAgentToken (agent // runtime tokens). Tests that override it affect both paths. @@ -223,8 +254,13 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep GitToken: composeGitToken, } - // Resolve agent source: config agents take precedence over disk harnesses. - harnessPath, fetchDeps, err := resolveAgentSource(ctx, absFullsendDir, agentName, orgCfg, composeOpts, printer) + // Resolve agent source: config agents take precedence, then agents repo + // fallback, then disk harnesses. + var fallbackForgeClient forge.Client + if composeGitToken != "" { + fallbackForgeClient = gh.New(composeGitToken) + } + harnessPath, fetchDeps, err := resolveAgentSource(ctx, absFullsendDir, agentName, fallbackForgeClient, orgCfg, composeOpts, printer) if err != nil { return err } @@ -2556,11 +2592,15 @@ func validateRepoNames(repos []string) error { } // resolveAgentSource resolves the harness path for an agent, checking -// config-registered agents before falling back to disk-based lookup. +// config-registered agents first, then falling back to the agents repo +// (fullsend-ai/agents), then to disk-based lookup. // Returns the local filesystem path to the harness (cached for URL sources) // and any fetch dependencies from URL-based agent resolution. -func resolveAgentSource(ctx context.Context, fullsendDir, agentName string, orgCfg *config.OrgConfig, composeOpts harness.ComposeOpts, printer *ui.Printer) (string, []harness.Dependency, error) { +func resolveAgentSource(ctx context.Context, fullsendDir, agentName string, forgeClient forge.Client, orgCfg *config.OrgConfig, composeOpts harness.ComposeOpts, printer *ui.Printer) (string, []harness.Dependency, error) { if orgCfg == nil || len(orgCfg.Agents) == 0 { + if path, deps, ok := tryAgentsRepoFallback(ctx, agentName, forgeClient, composeOpts, printer); ok { + return path, deps, nil + } path, err := resolveHarnessPath(fullsendDir, agentName, printer) return path, nil, err } @@ -2585,6 +2625,9 @@ func resolveAgentSource(ctx context.Context, fullsendDir, agentName string, orgC agent := config.LookupMergedAgent(merged, agentName) if agent == nil || !agent.IsConfig { + if path, deps, ok := tryAgentsRepoFallback(ctx, agentName, forgeClient, composeOpts, printer); ok { + return path, deps, nil + } path, err := resolveHarnessPath(fullsendDir, agentName, printer) return path, nil, err } @@ -2611,6 +2654,104 @@ func resolveAgentSource(ctx context.Context, fullsendDir, agentName string, orgC return contained, nil, nil } +// tryAgentsRepoFallback attempts to resolve an agent from the default agents +// repository (fullsend-ai/agents) by fetching the latest harness from the +// main branch. This is a transitional mechanism to support the extraction of +// first-party agents into a separate repository (fullsend-ai/agents) without +// requiring config changes from existing users. +// +// Returns (path, deps, true) on success, or ("", nil, false) if the fallback +// should be skipped (offline, no forge client, agent not known, not allowlisted, etc.). +// All errors are non-fatal — the caller falls through to disk-based lookup. +func tryAgentsRepoFallback(ctx context.Context, agentName string, forgeClient forge.Client, composeOpts harness.ComposeOpts, printer *ui.Printer) (string, []harness.Dependency, bool) { + normalizedName := strings.ToLower(agentName) + if !defaultAgentsRepoKnownAgents[normalizedName] { + return "", nil, false + } + if composeOpts.FetchPolicy.Offline { + return "", nil, false + } + if forgeClient == nil { + return "", nil, false + } + + allowlist := composeOpts.OrgAllowlist + if allowlist == nil { + allowlist = config.DefaultAllowedRemoteResources() + } + + branchSHA, err := forgeClient.GetBranchRef(ctx, defaultAgentsRepoOwner, defaultAgentsRepoName, defaultAgentsRepoBranch) + if err != nil { + printer.StepWarn(fmt.Sprintf("Could not resolve %s/%s@%s: %v", defaultAgentsRepoOwner, defaultAgentsRepoName, defaultAgentsRepoBranch, err)) + return "", nil, false + } + if !validHexSHA.MatchString(branchSHA) { + printer.StepWarn(fmt.Sprintf("Invalid branch SHA from %s/%s: %q", defaultAgentsRepoOwner, defaultAgentsRepoName, branchSHA)) + return "", nil, false + } + + rawURL := defaultAgentsRepoURLPrefix + branchSHA + "/harness/" + normalizedName + ".yaml" + + if harness.MatchingAllowedPrefixInList(rawURL, allowlist) == "" { + printer.StepWarn(fmt.Sprintf("Agents repo fallback skipped for %s: URL not in allowed_remote_resources", agentName)) + return "", nil, false + } + + shortSHA := branchSHA + if len(shortSHA) > 12 { + shortSHA = shortSHA[:12] + } + printer.StepStart(fmt.Sprintf("Fetching agent %s from %s/%s@%s", agentName, defaultAgentsRepoOwner, defaultAgentsRepoName, shortSHA)) + + content, err := fetch.FetchURL(ctx, rawURL, composeOpts.FetchPolicy) + if err != nil { + return "", nil, false + } + + // Content is fetched once and used directly — no self-referential hash + // verification. Supply-chain integrity relies on the commit-pinned URL, + // TLS transport, and the org allowlist. Config-registered agents get + // stronger pinning because their hashes are set at enrollment time. + contentHash := fetch.ComputeSHA256(content) + + if err := fetch.CachePut(composeOpts.WorkspaceRoot, rawURL, content); err != nil { + printer.StepWarn(fmt.Sprintf("Failed to cache agents repo content: %v", err)) + return "", nil, false + } + + cachePath, err := fetch.CachePath(composeOpts.WorkspaceRoot, contentHash) + if err != nil { + return "", nil, false + } + localPath := filepath.Join(cachePath, "content") + + if composeOpts.AuditLogPath != "" { + if err := fetch.AppendFetchAudit(composeOpts.AuditLogPath, fetch.FetchAuditEntry{ + TraceID: composeOpts.TraceID, + FetchTime: time.Now().UTC(), + URL: rawURL, + SHA256: contentHash, + FetchType: "static", + AllowedBy: harness.MatchingAllowedPrefixInList(rawURL, allowlist), + CacheHit: false, + }); err != nil { + printer.StepWarn(fmt.Sprintf("Failed to write fetch audit log: %v", err)) + } + } + + dep := harness.Dependency{ + Field: "base", + URL: rawURL, + LocalPath: localPath, + SHA256: contentHash, + FetchedAt: time.Now().UTC(), + Type: "file", + } + + printer.StepDone(fmt.Sprintf("Agent %s resolved from %s/%s (latest)", agentName, defaultAgentsRepoOwner, defaultAgentsRepoName)) + return localPath, []harness.Dependency{dep}, true +} + // containedLocalPath resolves a relative source path against baseDir and // verifies the result stays within baseDir. Returns an error for absolute // paths or paths that escape via traversal. diff --git a/internal/cli/run_test.go b/internal/cli/run_test.go index f3d9cf229..6eb15e065 100644 --- a/internal/cli/run_test.go +++ b/internal/cli/run_test.go @@ -6,6 +6,7 @@ import ( "encoding/json" "fmt" "io" + "net" "net/http" "net/http/httptest" "os" @@ -24,6 +25,7 @@ import ( "github.com/fullsend-ai/fullsend/internal/config" "github.com/fullsend-ai/fullsend/internal/fetch" "github.com/fullsend-ai/fullsend/internal/fetchsvc" + "github.com/fullsend-ai/fullsend/internal/forge" "github.com/fullsend-ai/fullsend/internal/harness" "github.com/fullsend-ai/fullsend/internal/mintclient" "github.com/fullsend-ai/fullsend/internal/ui" @@ -636,7 +638,7 @@ func TestResolveAgentSource_NoConfig(t *testing.T) { )) printer := ui.New(io.Discard) - path, deps, err := resolveAgentSource(context.Background(), dir, "code", nil, harness.ComposeOpts{}, printer) + path, deps, err := resolveAgentSource(context.Background(), dir, "code", nil, nil, harness.ComposeOpts{}, printer) require.NoError(t, err) assert.Contains(t, path, "code.yaml") assert.Empty(t, deps) @@ -659,7 +661,7 @@ func TestResolveAgentSource_ConfigLocalPath(t *testing.T) { } printer := ui.New(io.Discard) - path, deps, err := resolveAgentSource(context.Background(), dir, "custom", orgCfg, harness.ComposeOpts{}, printer) + path, deps, err := resolveAgentSource(context.Background(), dir, "custom", nil, orgCfg, harness.ComposeOpts{}, printer) require.NoError(t, err) assert.Equal(t, filepath.Join(dir, "harness", "custom.yaml"), path) assert.Empty(t, deps) @@ -677,7 +679,7 @@ func TestResolveAgentSource_ConfigLocalPathNotFound(t *testing.T) { } printer := ui.New(io.Discard) - _, _, err := resolveAgentSource(context.Background(), dir, "missing", orgCfg, harness.ComposeOpts{}, printer) + _, _, err := resolveAgentSource(context.Background(), dir, "missing", nil, orgCfg, harness.ComposeOpts{}, printer) require.Error(t, err) assert.Contains(t, err.Error(), "config agent missing") } @@ -693,7 +695,7 @@ func TestResolveAgentSource_ConfigLocalPathAbsoluteRejected(t *testing.T) { } printer := ui.New(io.Discard) - _, _, err := resolveAgentSource(context.Background(), dir, "evil", orgCfg, harness.ComposeOpts{}, printer) + _, _, err := resolveAgentSource(context.Background(), dir, "evil", nil, orgCfg, harness.ComposeOpts{}, printer) require.Error(t, err) assert.Contains(t, err.Error(), "absolute paths") } @@ -709,11 +711,330 @@ func TestResolveAgentSource_ConfigLocalPathTraversalRejected(t *testing.T) { } printer := ui.New(io.Discard) - _, _, err := resolveAgentSource(context.Background(), dir, "passwd", orgCfg, harness.ComposeOpts{}, printer) + _, _, err := resolveAgentSource(context.Background(), dir, "passwd", nil, orgCfg, harness.ComposeOpts{}, printer) require.Error(t, err) assert.Contains(t, err.Error(), "path traversal") } +func TestResolveAgentSource_AgentsRepoFallback_UnknownAgent(t *testing.T) { + dir := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(dir, "harness"), 0o755)) + + fakeClient := forge.NewFakeClient() + printer := ui.New(io.Discard) + _, _, err := resolveAgentSource(context.Background(), dir, "nonexistent", fakeClient, nil, harness.ComposeOpts{}, printer) + require.Error(t, err) + assert.Contains(t, err.Error(), "harness file not found") +} + +func TestResolveAgentSource_AgentsRepoFallback_Offline(t *testing.T) { + dir := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(dir, "harness"), 0o755)) + + fakeClient := forge.NewFakeClient() + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{ + FetchPolicy: fetch.FetchPolicy{Offline: true}, + } + _, _, err := resolveAgentSource(context.Background(), dir, "triage", fakeClient, nil, opts, printer) + require.Error(t, err) + assert.Contains(t, err.Error(), "harness file not found") +} + +func TestResolveAgentSource_AgentsRepoFallback_NoClient(t *testing.T) { + dir := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(dir, "harness"), 0o755)) + + printer := ui.New(io.Discard) + _, _, err := resolveAgentSource(context.Background(), dir, "triage", nil, nil, harness.ComposeOpts{}, printer) + require.Error(t, err) + assert.Contains(t, err.Error(), "harness file not found") +} + +func TestResolveAgentSource_AgentsRepoFallback_DiskFallback(t *testing.T) { + dir := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(dir, "harness"), 0o755)) + require.NoError(t, os.WriteFile( + filepath.Join(dir, "harness", "triage.yaml"), + []byte("agent: agents/triage.md\nrole: test\n"), + 0o644, + )) + + printer := ui.New(io.Discard) + path, deps, err := resolveAgentSource(context.Background(), dir, "triage", nil, nil, harness.ComposeOpts{}, printer) + require.NoError(t, err) + assert.Contains(t, path, "triage.yaml") + assert.Empty(t, deps) +} + +func TestTryAgentsRepoFallback_UnknownAgent(t *testing.T) { + fakeClient := forge.NewFakeClient() + printer := ui.New(io.Discard) + _, _, ok := tryAgentsRepoFallback(context.Background(), "custom-agent", fakeClient, harness.ComposeOpts{}, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_Offline(t *testing.T) { + fakeClient := forge.NewFakeClient() + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{FetchPolicy: fetch.FetchPolicy{Offline: true}} + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, opts, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_NilClient(t *testing.T) { + printer := ui.New(io.Discard) + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", nil, harness.ComposeOpts{}, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_GetBranchRefError(t *testing.T) { + fakeClient := forge.NewFakeClient() + fakeClient.Errors["GetBranchRef"] = fmt.Errorf("rate limited") + printer := ui.New(io.Discard) + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, harness.ComposeOpts{}, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_NotAllowlisted(t *testing.T) { + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = "abc123def456789012345678901234567890abcd" + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{ + OrgAllowlist: []string{"https://example.com/"}, + } + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, opts, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_ExplicitlyEmptyAllowlist(t *testing.T) { + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = "abc123def456789012345678901234567890abcd" + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{ + OrgAllowlist: []string{}, + } + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, opts, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_CaseNormalization(t *testing.T) { + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = "abc123def456789012345678901234567890abcd" + printer := ui.New(io.Discard) + + // "Triage" should pass the known-agent check but would have caused a 404 + // before the case-normalization fix. Now it uses "triage" in the URL. + _, _, ok := tryAgentsRepoFallback(context.Background(), "Triage", fakeClient, harness.ComposeOpts{}, printer) + // Fallback skips because fetch fails (no HTTP server), but it shouldn't + // panic and should get past the known-agent gate. + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_ShortSHA(t *testing.T) { + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = "abc" + printer := ui.New(io.Discard) + + // Short SHA fails hex validation — exercises both validation and bounds guard. + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, harness.ComposeOpts{}, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_InvalidSHA(t *testing.T) { + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" + printer := ui.New(io.Discard) + + // Non-hex characters should be rejected by SHA validation. + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, harness.ComposeOpts{}, printer) + assert.False(t, ok) +} + +func TestTryAgentsRepoFallback_AllKnownAgents(t *testing.T) { + for _, name := range []string{"triage", "code", "fix", "review", "retro", "prioritize"} { + t.Run(name, func(t *testing.T) { + fakeClient := forge.NewFakeClient() + printer := ui.New(io.Discard) + // Should pass the known-agent gate (not return false immediately). + // GetBranchRef will fail since no ref is set, confirming we got past the gate. + _, _, ok := tryAgentsRepoFallback(context.Background(), name, fakeClient, harness.ComposeOpts{}, printer) + assert.False(t, ok) + }) + } +} + +func TestTryAgentsRepoFallback_SuccessPath(t *testing.T) { + harnessContent := []byte("agent: agents/triage.md\nrole: test\n") + fakeSHA := "abcdef1234567890abcdef1234567890abcdef12" + + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + expectedPath := "/" + fakeSHA + "/harness/triage.yaml" + if r.URL.Path == expectedPath { + w.WriteHeader(http.StatusOK) + _, _ = w.Write(harnessContent) + } else { + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + + hostPort := strings.TrimPrefix(srv.URL, "https://") + hostname, port, _ := net.SplitHostPort(hostPort) + + tlsCfg := srv.TLS.Clone() + tlsCfg.InsecureSkipVerify = true + policy := fetch.NewTestPolicy(tlsCfg, []string{hostname}, []string{port}) + + orig := defaultAgentsRepoURLPrefix + defaultAgentsRepoURLPrefix = srv.URL + "/" + t.Cleanup(func() { defaultAgentsRepoURLPrefix = orig }) + + workDir := t.TempDir() + + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = fakeSHA + + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{ + WorkspaceRoot: workDir, + FetchPolicy: policy, + OrgAllowlist: []string{srv.URL + "/"}, + } + + path, deps, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, opts, printer) + require.True(t, ok, "expected fallback to succeed") + assert.NotEmpty(t, path) + assert.Len(t, deps, 1) + assert.Contains(t, deps[0].URL, fakeSHA) + assert.Equal(t, "file", deps[0].Type) + assert.NotEmpty(t, deps[0].SHA256) +} + +func TestTryAgentsRepoFallback_AuditLog(t *testing.T) { + harnessContent := []byte("agent: agents/triage.md\nrole: test\n") + fakeSHA := "abcdef1234567890abcdef1234567890abcdef12" + + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + expectedPath := "/" + fakeSHA + "/harness/triage.yaml" + if r.URL.Path == expectedPath { + w.WriteHeader(http.StatusOK) + _, _ = w.Write(harnessContent) + } else { + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + + hostPort := strings.TrimPrefix(srv.URL, "https://") + hostname, port, _ := net.SplitHostPort(hostPort) + + tlsCfg := srv.TLS.Clone() + tlsCfg.InsecureSkipVerify = true + policy := fetch.NewTestPolicy(tlsCfg, []string{hostname}, []string{port}) + + orig := defaultAgentsRepoURLPrefix + defaultAgentsRepoURLPrefix = srv.URL + "/" + t.Cleanup(func() { defaultAgentsRepoURLPrefix = orig }) + + workDir := t.TempDir() + auditLog := filepath.Join(workDir, "audit.jsonl") + + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = fakeSHA + + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{ + WorkspaceRoot: workDir, + FetchPolicy: policy, + OrgAllowlist: []string{srv.URL + "/"}, + AuditLogPath: auditLog, + TraceID: "test-trace-123", + } + + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, opts, printer) + require.True(t, ok, "expected fallback to succeed") + + auditContent, err := os.ReadFile(auditLog) + require.NoError(t, err) + assert.Contains(t, string(auditContent), fakeSHA) + assert.Contains(t, string(auditContent), "test-trace-123") + assert.Contains(t, string(auditContent), `"fetch_type":"static"`) +} + +func TestTryAgentsRepoFallback_CachePutFailure(t *testing.T) { + harnessContent := []byte("agent: agents/triage.md\nrole: test\n") + fakeSHA := "abcdef1234567890abcdef1234567890abcdef12" + + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + expectedPath := "/" + fakeSHA + "/harness/triage.yaml" + if r.URL.Path == expectedPath { + w.WriteHeader(http.StatusOK) + _, _ = w.Write(harnessContent) + } else { + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + + hostPort := strings.TrimPrefix(srv.URL, "https://") + hostname, port, _ := net.SplitHostPort(hostPort) + + tlsCfg := srv.TLS.Clone() + tlsCfg.InsecureSkipVerify = true + policy := fetch.NewTestPolicy(tlsCfg, []string{hostname}, []string{port}) + + orig := defaultAgentsRepoURLPrefix + defaultAgentsRepoURLPrefix = srv.URL + "/" + t.Cleanup(func() { defaultAgentsRepoURLPrefix = orig }) + + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = fakeSHA + + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{ + WorkspaceRoot: "/nonexistent/path/that/will/fail", + FetchPolicy: policy, + OrgAllowlist: []string{srv.URL + "/"}, + } + + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, opts, printer) + assert.False(t, ok, "expected fallback to fail when cache write fails") +} + +func TestTryAgentsRepoFallback_FetchURLError(t *testing.T) { + fakeSHA := "abcdef1234567890abcdef1234567890abcdef12" + + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusNotFound) + })) + t.Cleanup(srv.Close) + + hostPort := strings.TrimPrefix(srv.URL, "https://") + hostname, port, _ := net.SplitHostPort(hostPort) + + tlsCfg := srv.TLS.Clone() + tlsCfg.InsecureSkipVerify = true + policy := fetch.NewTestPolicy(tlsCfg, []string{hostname}, []string{port}) + + orig := defaultAgentsRepoURLPrefix + defaultAgentsRepoURLPrefix = srv.URL + "/" + t.Cleanup(func() { defaultAgentsRepoURLPrefix = orig }) + + fakeClient := forge.NewFakeClient() + fakeClient.BranchRefs["fullsend-ai/agents/main"] = fakeSHA + + printer := ui.New(io.Discard) + opts := harness.ComposeOpts{ + WorkspaceRoot: t.TempDir(), + FetchPolicy: policy, + OrgAllowlist: []string{srv.URL + "/"}, + } + + _, _, ok := tryAgentsRepoFallback(context.Background(), "triage", fakeClient, opts, printer) + assert.False(t, ok) +} + func TestContainedLocalPath_Valid(t *testing.T) { dir := t.TempDir() require.NoError(t, os.MkdirAll(filepath.Join(dir, "harness"), 0o755))