[OPIK-7940] [BE] [FE] feat: dynamic token auth (OAuth2 client credentials) for custom AI providers

[miguelgrc]

Details

Enterprise AI gateways commonly forbid static API keys and instead require short-lived bearer tokens minted via the OAuth2 client credentials grant. This PR adds a dynamic token auth mode to Custom AI providers (and Bedrock/Ollama-style custom endpoints): Opik fetches the token from the customer's auth service before LLM calls and manages its whole lifecycle, so online evaluations and the Playground work against these gateways with zero manual token handling.

• Storage (BE): new auth_config recipe (token URL, send mode, credentials, reply field paths, fallback TTL) stored AES-GCM-encrypted in one MySQL column. Secret values are write-only: read-back masks them with a __SECRET__ sentinel, and whole-document updates resolve sentinels against the stored recipe.
• Token lifecycle (BE): Redis-cached bearer shared across pods (cache key includes a recipe hash, so any config edit is an instant cache miss), proportional refresh ahead of expiry, cross-pod single-flight via distributed lock, degradation to direct fetch on Redis failure, and a retry-once path when the gateway rejects a token (revocation/clock skew). Streaming requests retry only before the first delivered event.
• Safety (BE): outbound token fetches go through a destination guard (https-only, resolve-then-decide, refuses loopback/link-local/private ranges) — strict by default, with relaxed shipped explicitly in the docker-compose and Helm distributions. OTel metrics cover token requests and fetch durations, labeled by outcome and origin (request vs test button).
• Test connection (BE+FE): POST /v1/private/llm-provider-key/auth-config/test runs the fetch server-side (rate-limited, redacted errors) and the dialog surfaces the result with the token lifetime.
• FE: the provider dialog gains an Authentication card with a "Static API key" / "OAuth2 client credentials" mode switch. The OAuth2 mode exposes only the spec's fields (token URL, client_id, client_secret, optional extras like scope); grant_type=client_credentials is injected automatically. Stored secrets render write-only (no reveal, locked). The backend recipe stays fully general, so other flows can be surfaced later as UI-only changes.

Change checklist

• User facing
• Documentation update

Issues

• Resolves OPIK-7940

AI-WATERMARK

AI-WATERMARK: yes

• If yes:
  • Tools: Claude Code
  • Model(s): Claude Fable 5
  • Scope: full implementation (BE + FE), built and reviewed part-by-part
  • Human verification: per-part code review of every diff, plus manual end-to-end testing against a live local stack

Testing

Automated (all passing locally):

• BE: mvn test on the feature's classes — token fetcher/cache (16), intercepting HTTP client incl. streaming retry semantics (14), destination guard (19), provider key resource incl. auth-config CRUD, masking and check endpoint (13), recipe validation and encryption utils. mvn compile + spotless:check clean after rebase on latest main.
• FE: npm run lint, npm run typecheck, npm run deps:validate, and 27 vitest cases over the form↔recipe converters (sentinel round-trip, clear-vs-omit semantics, grant_type injection/hiding).

Manual, against a local stack with a mock OAuth2 token service + bearer-validating gateway (and separately a real JWT-issuing OAuth2 server):

• Configure + Test connection (success and failure toasts, redacted error messages)
• Sentinel round-trip: reopen dialog, write-only secret, update without re-entering it
• Playground streaming through the fetched bearer
• Proactive refresh inside the TTL window; reactive 401 → invalidate → refetch → transparent retry; token expiry mid-stream (stream completes, no spurious re-auth)
• Destination guard: strict mode refuses local endpoints with an actionable message; relaxed override works

Not run: full repository-wide BE test suite (targeted classes instead, per repo convention). Playwright e2e specs for these flows are ready as a follow-up stacked PR.

Documentation

N/A — a docs page (provider setup + API-driven rotation snippet) follows in the stacked e2e/docs PR.

[github-actions]

🔄 Test environment deployment process has started

Phase 1: Deploying base version 2.2.32-6370 (from main branch) if environment doesn't exist
Phase 2: Building new images from PR branch miguelg/OPIK-7940-dynamic-token-auth
Phase 3: Will deploy newly built version after build completes

You can monitor the progress here.

[github-actions]

⏱️ pre-commit per-hook timing

Hook	Description	Result	Duration
🌐 typecheck — frontend	Whole-project tsc type check	✅	29.61s
☕ spotless — java backend	Format Java code	✅	6.25s
🌐 eslint — frontend	Lint + autofix JS/TS	✅	4.97s
⚓ helm-docs	Regenerate Helm chart README	✅	4.75s
Total (4 ran)			45.58s

⏭️ 39 skipped (no matching files changed)

Hook	Description	Result
🐍 trim trailing whitespace — python sdk	Strip trailing whitespace	⏭️
🐍 fix end of files — python sdk	Ensure files end in a newline	⏭️
🐍 ruff — python sdk	Lint + autofix Python (ruff)	⏭️
🐍 ruff-format — python sdk	Format Python code (ruff)	⏭️
🐍 mypy — python sdk	Static type check	⏭️
🤖 trim trailing whitespace — optimizer	Strip trailing whitespace	⏭️
🤖 fix end of files — optimizer	Ensure files end in a newline	⏭️
🤖 check yaml — optimizer	Validate YAML syntax	⏭️
🤖 check json — optimizer	Validate JSON syntax	⏭️
🤖 check toml — optimizer	Validate TOML syntax	⏭️
🤖 check for added large files — optimizer	Block large files (>1MB)	⏭️
🔐 detect private key — optimizer	Block committed private keys	⏭️
🤖 check for merge conflicts — optimizer	Block merge-conflict markers	⏭️
🤖 check for case conflicts — optimizer	Block case-only name clashes	⏭️
🤖 pyupgrade — optimizer	Modernize Python syntax	⏭️
🤖 ruff — optimizer	Lint + autofix Python (ruff)	⏭️
🤖 ruff-format — optimizer	Format Python code (ruff)	⏭️
🤖 mypy — optimizer	Static type check	⏭️
📓 nbstripout — optimizer notebooks	Strip notebook output	⏭️
📝 markdownlint — optimizer	Lint Markdown	⏭️
🔤 codespell — optimizer	Fix common misspellings	⏭️
📊 radon cc — optimizer	Cyclomatic-complexity gate	⏭️
📊 radon raw — optimizer	Raw size metrics gate	⏭️
📊 xenon — optimizer	Fail on complexity thresholds	⏭️
📊 lizard — optimizer	Cyclomatic-complexity gate	⏭️
🧹 vulture — optimizer	Find dead code	⏭️
🛡️ trim trailing whitespace — guardrails	Strip trailing whitespace	⏭️
🛡️ fix end of files — guardrails	Ensure files end in a newline	⏭️
🛡️ ruff — guardrails	Lint + autofix Python (ruff)	⏭️
🛡️ ruff-format — guardrails	Format Python code (ruff)	⏭️
🛡️ mypy — guardrails	Static type check	⏭️
block non-public FE plugins	Block non-public FE plugins	⏭️
🧪 pre-commit wrapper smoke tests	Self-test the wrapper scripts	⏭️
🧪 rebaseline script tests	Self-test the changelog re-baseline script	⏭️
📘 eslint — typescript sdk	Lint + autofix JS/TS	⏭️
📘 typecheck — typescript sdk	Whole-project tsc type check	⏭️
⚙️ actionlint — github workflows	Lint GitHub Actions workflows	⏭️
🐳 hadolint — dockerfiles	Lint Dockerfiles	⏭️
🌈 zizmor — github workflows security	Security-scan GitHub Actions workflows	⏭️

[CometActions]

This change looks worth a test.

This adds an OAuth2 client-credentials auth mode to the custom-provider dialog (new Authentication card, credential rows, Check connection), a new POST /v1/private/llm-provider-key/auth-config/test, and a write-only SECRET round-trip on GET — and it moves the existing static API key field inside that new card. Nothing e2e asserts this dialog today: configuration.ai-provider-add and ai-provider-edit are both covered:false, playground-providers.spec.ts (@provider-sanity, skipped without real provider keys) drives the add form but only asserts a row appears, and visual C03 just screenshots the providers table. So a secret sentinel saved literally on edit, or api_key and token mode not staying mutually exclusive, would ship silently. It's reachable with no provider keys — docker-compose sets LLM_PROVIDER_TOKEN_AUTH_DESTINATION_GUARD=relaxed, so a local mock token endpoint is allowed.

Would target configuration.ai-provider-add, configuration.ai-provider-edit.

What it would check

1. Add a custom (OpenAI-compatible) provider in OAuth2 client credentials mode with a token URL and client_id/client_secret/scope rows; save, reopen the dialog and GET the key, and confirm secret rows read back locked as SECRET while scope reads back in clear
2. Edit that provider (change scope or the models list) without retyping the secret, save, and confirm the stored client_secret still works rather than being overwritten with the literal sentinel
3. Switch the same provider between Static API key and OAuth2 mode and confirm the two are mutually exclusive on the saved record — saving in token mode clears api_key, switching back drops auth_config
4. Point Check connection at a mock token endpoint returning {access_token, expires_in} and confirm the success toast reports the lifetime; then point it at a 401/unreachable URL and confirm a destructive toast with the backend message and no token leaked in the response
5. Regression pass on the unchanged path: add and edit a plain static-API-key custom provider (API key, auth header name, suppress-default-auth now live inside the new Authentication card) and confirm it still saves and appears in the providers table

Deploying a test environment for this PR and exploring it — results will follow in a comment.

also touches Backend (Java API / internal), Deployment / Helm
Run

Advisory, from the QA test radar. Nothing here blocks this PR, and anything it proposes is a draft for review.

Re-checked after a push on 19 Aug 15:36 UTC.

[CometActions]

✅ Test environment is now available!

To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml)

Access Information

• URL: https://pr-7910.dev.comet.com
• Cluster: comet-ml-development
• Namespace: pr-7910
• Version: 2.2.32-7910-merge-3020
• Application logs: View in Grafana

The deployment has completed successfully and the version has been verified.

[CometActions]

🌙 Nightly cleanup: The test environment for this PR (pr-7910) has been cleaned up to free cluster resources. PVCs are preserved — re-deploy to restore the environment.