Skip to content

thv llm setup doesn't append provider path prefix for Envoy AI Gateway #5158

@jerm-dro

Description

@jerm-dro

Problem

thv llm setup writes ANTHROPIC_BASE_URL=<gateway-url> verbatim. When the gateway is an Envoy AI Gateway, native-Anthropic traffic is served at /anthropic/v1/messages, not /v1/messages. Without the prefix, Claude Code's requests fall through to a catchall route and the user sees:

There's an issue with the selected model (claude-sonnet-4-6). It may not exist or you may not have access to it.

Gateway logs show 404, model: null, with auth headers populated — i.e. auth works, the path doesn't.

Workaround

After running thv llm setup, manually append /anthropic to the gateway URL in the patched tool config (e.g. set ANTHROPIC_BASE_URL=https://<gateway>/anthropic in Claude Code's settings).

What needs to change

thv llm setup should know that some gateways prefix provider APIs. Two pieces:

  1. --anthropic-path-prefix flag (default empty). Users on Envoy AI Gateway pass --anthropic-path-prefix=/anthropic; others leave it.
  2. Optional auto-probe. HEAD/OPTIONS <gateway>/anthropic/v1/messages vs /v1/messages and pick the one that returns 401 (auth-reachable) over 404.

Scope is direct-mode tools only (Claude Code, Codex). Proxy-mode tools (Cursor, VS Code) route through thv llm proxy and don't need the prefix in their config.

Acceptance

  • --anthropic-path-prefix flag exists and is honored when patching ANTHROPIC_BASE_URL.
  • Setup against an Envoy AI Gateway with the flag set produces a working Claude Code config.
  • Setup against a non-prefixed gateway (LiteLLM, direct Anthropic) still works with the flag unset.

Metadata

Metadata

Assignees

Labels

cliChanges that impact CLI functionalityenhancementNew feature or requestgoPull requests that update go codellm gatewayLLM gateway authentication feature

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions