feat(BA-5528): add deployment chat CLI for vLLM-backed model services

[jopemachine]

📚 Stacked PRs

This PR is part of a 2-PR stack. Merge in order:

1. 👉 feat(BA-5528): add deployment chat CLI for vLLM-backed model services #11344 — feat(BA-5528): add deployment chat CLI for vLLM-backed model services ← you are here
2. ⬇️ feat(BA-5903): persist deployment chat history and replay as request context #11412 — feat(BA-5903): persist deployment chat history and replay as request context

Summary

• Add ./bai deployment chat <id> "<content>" for one-shot OpenAI-compatible chat against deployed inference services. Requests are sent directly to the deployment's inference endpoint with optional Authorization: Bearer <token> (the value the runtime — vLLM/SGLang/NIM/TGI/custom — was started with), bypassing the Backend.AI manager. Use --params to forward runtime-variant-specific sampling knobs.
• Add ./bai deployment chat-config set/show/clear/clear-cache to register, inspect, and remove per-deployment chat state.
• Auto-derive the request model when the user did not specify one: the CLI calls GET /v1/models on the inference endpoint, picks data[0].id, and caches it as cache.default_model for subsequent calls (matches the webui ChatCard.tsx fallback). The user is no longer required to run chat-config set --model before the first chat.
• Persist state under ~/.backend.ai/deployment_chat/, grouped per-feature (matching the existing ~/.backend.ai/session/ layout used by ./bai login):
  • cache.json — auto-managed: endpoint_url, default_model (auto-fetched from /v1/models), last_synced_at (24-h TTL).
  • config.json — user-managed: per-deployment { token, model } entries. The user's model takes precedence over cache.default_model.
• Both files are written via plain path.write_text() to match the existing CLI credential-storage convention (client/cli/v2/config_cmd.py). On 401/403 from the inference endpoint, the cached token for that deployment is cleared and the user is prompted to re-register.
• Add an SDK-side BackendAIAppProxyClient base in client/v2/base_client.py for direct-to-deployment HTTP traffic (Bearer-token auth, app-proxy-aware JSON parsing) and a thin DeploymentChatClient subclass exposing chat_completion() and list_models() (returning a typed ListModelsResponse).

Model resolution order

When the runtime needs a model field for a chat call, the CLI walks this list and stops at the first hit:

1. --model <name> on the chat command line.
2. config.<deployment-id>.model — the user's pinned model in config.json.
3. cache.<deployment-id>.default_model — the auto-derived value in cache.json.
4. GET /v1/models on the inference endpoint, taking data[0].id. The result is written to cache.default_model so subsequent calls skip the round trip.

This means a fresh deployment works with zero configuration as long as the runtime serves /v1/models; you only need chat-config set --model for multi-model deployments where [0] is not the right pick.

Command usage

# One-shot chat — model is auto-derived on first call from /v1/models
./bai deployment chat <deployment-id> "Hello, who are you?"

# Override the model for one call
./bai deployment chat <deployment-id> "..." --model llama-3-8b-instruct

# Forward runtime-specific sampling knobs as a JSON object
./bai deployment chat <deployment-id> "..." \
    --params '{"temperature": 0.7, "max_tokens": 256}'

# Register a Bearer token for a token-gated deployment
./bai deployment chat-config set <deployment-id> --token <runtime-token>

# Pin a model (overrides the cached default; useful for multi-model deployments)
./bai deployment chat-config set <deployment-id> --model llama-3-8b-instruct

# Set both at once
./bai deployment chat-config set <deployment-id> \
    --token <runtime-token> --model llama-3-8b-instruct

# Inspect what's currently registered/cached (token is masked)
./bai deployment chat-config show <deployment-id>

# Remove the user-managed config entry (token + model) for a deployment
./bai deployment chat-config clear <deployment-id>

# Force-invalidate the auto-managed cache entry (endpoint_url, default_model)
./bai deployment chat-config clear-cache <deployment-id>

chat-config set writes to config.json only — it does not contact the manager, so it stays usable while the deployment is still provisioning or the manager is unreachable. chat-config clear and clear-cache operate on the two storage files independently: clearing user config never touches the cache, and vice versa.

On-disk state

State lives under ~/.backend.ai/deployment_chat/ so it stays grouped with the other Backend.AI CLI state directories.

cache.json — auto-managed by the CLI; do not hand-edit.

{
  "deployments": {
    "d55e251a-3a70-408d-97a9-ca305502aa58": {
      "endpoint_url": "https://app-proxy.example.com/v1/some-deployment",
      "default_model": "llama-3-8b-instruct",
      "last_synced_at": "2026-04-29T12:34:56.789012+00:00"
    }
  }
}

• endpoint_url — fetched from the manager's deployment.network_access.endpoint_url and refreshed on a 24-hour TTL.
• default_model — auto-derived from GET /v1/models on first use; never written by chat-config set.
• last_synced_at — UTC timestamp of the last manager fetch; entries past CACHE_ENTRY_TTL (24 h) are treated as a cache miss.

config.json — user-managed: one { token, model } entry per deployment.

{
  "deployments": {
    "d55e251a-3a70-408d-97a9-ca305502aa58": {
      "token": "sk-runtime-token-here",
      "model": "llama-3-8b-instruct"
    }
  }
}

Either field may be null — chat-config set upserts only the fields you pass, and an entry is dropped automatically once both fields are cleared. The token is also cleared automatically on 401/403 from the inference endpoint so the next chat call surfaces the re-register hint instead of silently re-sending a stale credential.

Resolves BA-5528.

[Copilot]

Pull request overview

Adds a new CLI workflow for one-shot OpenAI-compatible chat calls against deployed vLLM inference endpoints, backed by a per-deployment local cache and a dedicated SDK-side HTTP client/DTOs (bypassing the Backend.AI manager API).

Changes:

• Add ./bai deployment chat and ./bai deployment chat-config set/show/clear commands plus a JSON cache at ~/.backend.ai/deployment_chat.json (0600, atomic write).
• Add DeploymentChatClient (direct aiohttp client) and OpenAI-compatible Pydantic DTOs under the v2 client package.
• Add unit tests for the direct chat client and the cache load/save semantics.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/unit/client/v2/test_deployment_chat_client.py	Unit tests for direct vLLM chat posting, auth error handling, serialization, and session ownership.
tests/unit/client/cli/test_deployment_chat_cache.py	Unit tests for cache schema/version guard, permissions, atomic write, masking, and tolerant loading.
src/ai/backend/client/v2/domains_v2/deployment_chat.py	New direct-to-inference chat client (aiohttp) with OpenAI-compatible request/response handling.
src/ai/backend/client/v2/chat_dto.py	New Pydantic DTOs for /v1/chat/completions request/response payloads with forward-compatible extra fields.
src/ai/backend/client/cli/v2/deployment_chat_cache.py	New cache implementation for endpoint URL + vLLM API key persistence with 0600 permissions and atomic writes.
src/ai/backend/client/cli/v2/deployment/chat_config.py	New chat-config CLI group to set/show/clear cache entries.
src/ai/backend/client/cli/v2/deployment/chat.py	New chat CLI command to send one-shot chat completions and invalidate cached key on 401/403.
src/ai/backend/client/cli/v2/deployment/init.py	Registers the new chat and chat-config commands under deployment.
changes/5528.feature.md	Changelog entry for the new CLI commands and cache behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.