diff --git a/skills/build-agent/SKILL.md b/skills/build-agent/SKILL.md index 54c1da6..d5a0692 100644 --- a/skills/build-agent/SKILL.md +++ b/skills/build-agent/SKILL.md @@ -34,6 +34,17 @@ self-hosts (it defaults to Agenta cloud, `https://cloud.agenta.ai`). - Ask the user for their API key. Point them to it: the **API keys** page in their Agenta project settings (on cloud, under `https://cloud.agenta.ai`). Ask for the host URL only if they self-host, in which case it is their own Agenta domain. +- **Ask which LLM connection mode they want** (`llm.connection.mode` in the config below). + `self_managed` means the *runner* itself is logged into its own Claude Code / Codex + subscription (Claude Pro/Max, ChatGPT Plus, etc.) — Agenta needs no provider API key at all + for this. `agenta` means the user provides their provider API key *in Agenta* (the project + vault) and Agenta uses that key on every call. **On Agenta cloud, `self_managed` is not + available at all** — cloud runners have no subscription of their own, so cloud always means + `agenta`, and you need the user's provider API key. On a self-hosted instance, either mode + can work, so ask which one applies: if the runner has its own Claude Code / Codex + subscription, use `self_managed` and skip the provider key question entirely; otherwise use + `agenta` and ask for the provider (`anthropic`, for the default `claude` harness — see the + coupling note below) and point them to that provider's key page. - Then offer to set it up for them, either way they prefer: - **Write it for them:** create a `.env` file in the working directory with `AGENTA_API_KEY=...` (and `AGENTA_HOST=...` if self-hosting). `lib.sh` picks up a local @@ -66,10 +77,19 @@ The rest is fixed boilerplate — copy it exactly: } ``` -Keep `llm` / `harness` / `runner` / `sandbox` exactly as above unless told otherwise. `model` -is an alias (`sonnet`, `opus`, `haiku`, `default`), never a raw model id. For the full config -field by field — tool entries, skill entries, and the fields that 500 if misplaced — read -`references/config-schema.md`. +Keep `llm` / `harness` / `runner` / `sandbox` exactly as above unless told otherwise, except +for `llm.connection.mode` (`self_managed` vs `agenta` — see the credentials step above) and +`llm.model` (an alias: `sonnet`, `opus`, `haiku`, `default`, never a raw model id). **This is +not one of the "fewest calls" simplifications** — do not improvise or substitute a different +provider here even for the simplest one-line agent. `harness.kind: "claude"` only runs +Anthropic models: `llm.provider` must stay `"anthropic"`. Pairing `harness.kind: "claude"` +with any other provider (e.g. an `"openai"`/raw-model-id config you might see elsewhere, such +as in the platform's own bare API schema default — that default is unrelated to this skill and +not compatible with this harness) is accepted by the create/commit endpoints with no error, but +the agent then gets no Model & Harness resolved in the Agenta UI and never runs correctly. +`create-agent.sh` / `update-agent.sh` now reject this mismatch before calling the API — if you +hit that error, fix the config rather than retrying. For the full config field by field — tool +entries, skill entries, and the fields that 500 if misplaced — read `references/config-schema.md`. ## The loop (run it in order, skip what the ask doesn't need) @@ -87,8 +107,8 @@ field by field — tool entries, skill entries, and the fields that 500 if mispl names the exact tools in order and ends on the terminal action — otherwise the run wanders or stops short. Depth in `references/writing-instructions.md`. 5. **Create and test in one shot:** `bash scripts/build.sh ""`. - Read the OUTPUT and RESOLVED lines. RESOLVED must show - `harness=claude model=sonnet connection=self_managed`. + Read the OUTPUT and RESOLVED lines. RESOLVED must show `harness=claude model=sonnet` and + the `connection` mode you configured (`self_managed` or `agenta`). 6. **If it needs a trigger**, create it against the variant id AND the revision id (both printed by `create-agent.sh` / `build.sh`) — a schedule with `bash scripts/create-schedule.sh`, an event subscription with `bash scripts/create-subscription.sh` (only after @@ -162,6 +182,15 @@ Keep to the loop above for a simple agent. Read one of these only when the task ## Hard rules (these prevent the usual failures) +- **`harness`/`runner`/`sandbox` are fixed, not a "fewest calls" shortcut; `llm.provider` is + fixed to the harness.** Copy them byte-for-byte from the boilerplate, even for the simplest + one-line agent — do not simplify or substitute a provider here. `harness.kind: "claude"` only + runs Anthropic models; if `llm.provider` is anything but `"anthropic"`, the agent gets no + Model & Harness resolved in the Agenta UI and silently never runs, even though creation + itself succeeds with no error. `create-agent.sh` / `update-agent.sh` reject this mismatch + before calling the API — treat that rejection as a config bug to fix, never as a reason to + retry with different values. `llm.connection.mode` is the one field in this block you DO + choose (see the credentials step): `self_managed` or `agenta`. - **A trigger needs the revision id, not just the variant id.** `create-schedule.sh` and `create-subscription.sh` both take ` ...` — pass both (printed by `create-agent.sh` / `build.sh`), or pass `latest` in place of the revision id if you only have diff --git a/skills/build-agent/references/config-schema.md b/skills/build-agent/references/config-schema.md index 582d5c3..e5a8266 100644 --- a/skills/build-agent/references/config-schema.md +++ b/skills/build-agent/references/config-schema.md @@ -48,12 +48,30 @@ Keep `llm`, `mcps`, `harness`, `runner`, and `sandbox` exactly as in the templat user asks otherwise. - `llm.model` is an **alias** — `sonnet`, `opus`, `haiku`, or `default` — never a raw model id. -- `llm.provider` is `anthropic` and `llm.connection.mode` is `self_managed` in the default - setup. +- `llm.provider` is `anthropic` in the default setup. `llm.connection.mode` is the one field in + this block you actually choose: `self_managed` (the *runner* is logged into its own Claude + Code / Codex subscription — Agenta needs no provider API key) or `agenta` (the user's + provider API key lives in Agenta's project vault and Agenta uses it on every call). **On + Agenta cloud, `self_managed` is not available at all** — cloud runners have no subscription + of their own, so cloud always means `agenta`. On a self-hosted instance, ask the user which + applies (`SKILL.md`'s credentials step) rather than assuming either one. - `harness.kind` is `claude`. `runner.kind` is `sidecar`. `sandbox.kind` is `local`. -RESOLVED (printed by `test-agent.sh`) must echo these back: -`harness=claude model=sonnet connection=self_managed`. If it does not, the run fell back to a +**`harness.kind` and `llm.provider` are coupled, not independent fields.** `harness.kind: +"claude"` only runs Anthropic models — `llm.provider` must stay `"anthropic"` whenever +`harness.kind` is `"claude"`. Don't change `llm.provider`/`llm.model` while leaving +`harness.kind: "claude"` in place, even if you've seen a different default somewhere else +(for example, the platform's own bare API schema for this endpoint shows a different +`provider`/`harness.kind` pairing entirely — that's the platform's raw baseline for building +agents *without* this skill, and it is not compatible with the `claude` harness this skill +uses). This mismatch is accepted by the create/commit endpoints with no server-side error, +but the agent then gets no Model & Harness resolved in the Agenta UI and never runs +correctly — `create-agent.sh` / `update-agent.sh` now check for it and refuse to proceed if +found, rather than let you learn this from a silently broken agent. `llm.connection.mode` is +not part of this coupling — either mode works with the `claude` harness. + +RESOLVED (printed by `test-agent.sh`) must echo these back: `harness=claude model=sonnet` and +the `connection` mode you configured. If the harness or model differ, the run fell back to a default — fix the config and re-test. ## A tool entry (gateway object) diff --git a/skills/build-agent/scripts/create-agent.sh b/skills/build-agent/scripts/create-agent.sh index 963a1cb..04d9e32 100755 --- a/skills/build-agent/scripts/create-agent.sh +++ b/skills/build-agent/scripts/create-agent.sh @@ -13,6 +13,8 @@ CONFIG_FILE="${1:?usage: create-agent.sh [name]}" SLUG="${2:?usage: create-agent.sh [name]}" NAME="${3:-$SLUG}" +validate_agent_config "$CONFIG_FILE" || exit 1 + AGENT_CONFIG="$(cat "$CONFIG_FILE")" BODY="$(jq -n --arg slug "$SLUG" --arg name "$NAME" --argjson agent "$AGENT_CONFIG" ' {application: {slug: $slug, name: $name, diff --git a/skills/build-agent/scripts/lib.sh b/skills/build-agent/scripts/lib.sh index 71f4e57..41731d8 100755 --- a/skills/build-agent/scripts/lib.sh +++ b/skills/build-agent/scripts/lib.sh @@ -43,6 +43,83 @@ agenta_post() { curl -s -X POST "$AGENTA_HOST$1" "${_AUTH[@]}" -d "$2"; } agenta_get() { curl -s "$AGENTA_HOST$1" "${_AUTH[@]}"; } agenta_delete() { curl -s -X DELETE "$AGENTA_HOST$1" "${_AUTH[@]}"; } +# Checks that harness.kind and llm.provider are a consistent pairing, when both are +# present in the given config file. The Claude Code harness ("claude") only runs +# Anthropic models — pairing it with any other provider is accepted by the create/commit +# endpoints with no server-side error, but the agent then gets no Model & Harness +# resolved in the Agenta UI and won't run correctly. This check is tolerant of missing +# fields (safe to use on a partial update-agent.sh delta, which merges onto the existing +# config) — it only fires when both fields are actually present and inconsistent. +# +# Usage: _check_harness_provider_consistency +_check_harness_provider_consistency() { + local config_file="$1" + local harness_kind provider model + harness_kind="$(jq -r '.harness.kind // empty' "$config_file" 2>/dev/null || true)" + provider="$(jq -r '.llm.provider // empty' "$config_file" 2>/dev/null || true)" + model="$(jq -r '.llm.model // empty' "$config_file" 2>/dev/null || true)" + + if [ -n "$harness_kind" ] && [ -n "$provider" ] && [ "$harness_kind" = "claude" ] && [ "$provider" != "anthropic" ]; then + { + echo "CONFIG ERROR: harness.kind=\"claude\" requires llm.provider=\"anthropic\" — the" + echo " Claude Code harness only runs Anthropic models. Found llm.provider=\"$provider\"," + echo " llm.model=\"$model\" instead." + echo " This exact mismatch is accepted by the API with no error, but the agent then" + echo " gets no Model & Harness resolved in the Agenta UI and won't run correctly." + echo " Fix: set llm.provider back to \"anthropic\" and llm.model to one of" + echo " sonnet/opus/haiku/default (see SKILL.md). This skill does not currently support" + echo " building agents on a non-Claude harness — do not substitute a different provider" + echo " while leaving harness.kind=\"claude\"." + } >&2 + return 1 + fi + return 0 +} + +# Full validation for a brand-new agent config (create-agent.sh). Requires the fixed +# boilerplate fields to all be present — a simple agent has no reason to omit any of +# them — plus the harness/provider consistency check above. +# +# Usage: validate_agent_config +# Prints nothing on success. On failure, prints a specific CONFIG ERROR to stderr and +# returns non-zero — the caller should stop and fix the config rather than proceed. +validate_agent_config() { + local config_file="$1" + local field val + + for field in '.llm.model' '.llm.provider' '.llm.connection.mode' \ + '.harness.kind' '.runner.kind' '.sandbox.kind'; do + val="$(jq -r "$field // empty" "$config_file" 2>/dev/null || true)" + if [ -z "$val" ]; then + { + echo "CONFIG ERROR: required field $field is missing or empty in $config_file." + echo " instructions / tools / mcps / skills are the only fields you write yourself —" + echo " llm / harness / runner / sandbox must be copied exactly from the boilerplate" + echo " in SKILL.md, even for the simplest agent. Fix the config before creating/testing." + } >&2 + return 1 + fi + done + + _check_harness_provider_consistency "$config_file" +} + +# Lighter validation for an update-agent.sh delta, which deep-merges onto the existing +# config and so may legitimately omit any of the boilerplate fields. Only checks for an +# internal inconsistency in what IS being submitted. +# +# LIMITATION: this does not fetch and merge the current committed config, so it only +# catches a mismatch when both harness.kind and llm.provider are present together in the +# delta itself. A delta that changes ONLY llm.provider (relying on merge to keep a +# previous harness.kind: "claude") will NOT be caught here. This is why update-agent.sh's +# own guidance is to always pass the complete config, boilerplate included — do that +# rather than relying on this check as a full safety net for a minimal delta. +# +# Usage: validate_agent_config_delta +validate_agent_config_delta() { + _check_harness_provider_consistency "$1" +} + # Resolves a argument for create-schedule.sh / create-subscription.sh. # A real revision id (from create-agent.sh / build.sh / update-agent.sh) passes straight # through. The literal value "latest" (case-insensitive) instead fetches the variant's diff --git a/skills/build-agent/scripts/update-agent.sh b/skills/build-agent/scripts/update-agent.sh index 7f5cb85..fc92891 100755 --- a/skills/build-agent/scripts/update-agent.sh +++ b/skills/build-agent/scripts/update-agent.sh @@ -18,6 +18,8 @@ VARIANT="${1:?usage: update-agent.sh [message]}" CONFIG_FILE="${2:?usage: update-agent.sh [message]}" MESSAGE="${3:-update agent config}" +validate_agent_config_delta "$CONFIG_FILE" || exit 1 + AGENT_CONFIG="$(cat "$CONFIG_FILE")" BODY="$(jq -n --arg vid "$VARIANT" --arg msg "$MESSAGE" --argjson agent "$AGENT_CONFIG" \ '{workflow_revision:{workflow_variant_id:$vid, message:$msg,