feat(#3647): auth pivot, dev scaffold, entity provider fixes (#3648)

* feat(#3647): auth pivot, dev scaffold, entity provider fixes

Pivot Kagenti authentication from per-user RFC 8693 token exchange
to service-account OAuth2 Client Credentials Grant. Add dev
scaffold (app-config.yaml, packages/backend, load-secrets.sh).
Fix entity provider API URLs and response handling.

Auth pivot (config.d.ts, schemas.ts, schemas.test.ts):
- Replace tokenExchange.{enabled,audience,userTokenHeader} with
  flat tokenEndpoint, clientId, clientSecret, tokenExpiryBufferSeconds
- Bump BOOST_CONFIG_SCHEMA_VERSION to 2
- clientSecret marked sensitive: true with @visibility secret
- tokenExpiryBufferSeconds: .number().int().min(0).optional()
- Add validation tests for new fields

Entity provider fixes (kagenti-entity-provider):
- New KeycloakAuthClient: OAuth2 Client Credentials Grant with
  token caching and configurable expiry buffer
- Fix API URLs from /a2a/ to /api/v1/ for agents and tools
- Add unwrapItems helper for {items:T[]} and T[] responses
- Pass optional authClient to entity providers
- Module reads boost.kagenti.auth.* and constructs auth client
- Add tests for auth, URL patterns, response unwrapping

Dev scaffold:
- app-config.yaml with guest auth, SQLite, dev providers
- packages/backend host with metrics shim and boost plugins
- scripts/load-secrets.sh for K8s dev secret loading

Spec updates across 14 specification/openspec files:
- Rename "Per-User Identity Delegation" to "Service-Account
  Keycloak Authentication" throughout
- Add boost-responses-api-toolkit and boost-toolscope to workspace
  structure docs
- Add Local Development section to boost-context.md
- Update staged-issues.md issue 13 and add issue 16 URL

TODO comments added to mcp/routes.ts and skills/routes.ts for
future catalog cross-reference.

Closes #3647

* fix(kagenti-entity-provider): address review findings for PR 3648

- Remove fail-open auth pattern: let getBearerToken() throw propagate
  to refresh() catch which serves cached entities (not unauthenticated)
- Wire tokenExpiryBufferSeconds from config to KeycloakAuthClient
- Clamp token expiry to prevent hot-loop when buffer >= token lifetime
- Validate access_token exists in Keycloak response before caching
- Remove dead auth field from KagentiEntityProviderConfig type
- Update tests to expect auth-failure-serves-cached behavior

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(boost): address review findings from second review agent run

- Add partial auth config detection with warning log in readKagentiAuthConfig
  when only 1-2 of 3 required fields (tokenEndpoint, clientId, clientSecret)
  are present
- Rename KeycloakTokenManager to KeycloakAuthClient across all docs and specs
  for naming consistency with the actual code class name
- Fix stale class name and file path in pluggable-ai-platform-architecture PRD
  cache migration table

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: gabemontero <gmontero@redhat.com>

* docs(boost): update issue 3309 and staged-issues with PR 3648 completions

Mark tasks 7.1, 7.2, 7.4, 7.5a as complete in tasks.md and
staged-issues.md. Split task 7.5 into 7.5a (entity providers, done)
and 7.5b (KagentiApiClient, remaining). Add note clarifying that
remaining auth tasks target boost-backend-module-kagenti.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(boost): add issue 17 (KeycloakAuthClient cacheService migration) to staged-issues

Tracks #3654 — refactor KeycloakAuthClient to use Backstage
cacheService instead of in-memory token cache, aligning with
design principle 1.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(boost): harden dev scaffold security defaults

- Remove dangerouslyAllowOutsideDevelopment from guest auth config;
  Backstage already allows guest auth in development mode by default
- Make NODE_TLS_REJECT_UNAUTHORIZED overridable in load-secrets.sh
  so operators can enforce TLS verification when not using self-signed
  certs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(boost): clarify X-Backstage-User scope in auth specs

X-Backstage-User header only applies to user-initiated requests via
KagentiApiClient (chat, agent operations), not entity provider
background polling which has no user context. Updated access-control
spec, security design doc, and boost-context to make this distinction
explicit.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: gabemontero <gmontero@redhat.com>

---------

Signed-off-by: gabemontero <gmontero@redhat.com>
Co-authored-by: fullsend-code <278716306+fullsend-ai-coder[bot]@users.noreply.github.com>
Co-authored-by: gabemontero <gmontero@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: 3 people

workspaces/boost/app-config.yaml

@@ -0,0 +1,64 @@
+app:
+  title: Boost Dev
+  baseUrl: http://localhost:3000
+
+organization:
+  name: Red Hat
+
+backend:
+  baseUrl: http://localhost:7007
+  listen:
+    port: 7007
+  database:
+    client: better-sqlite3
+    connection: ':memory:'
+  auth:
+    dangerouslyDisableDefaultAuthPolicy: true
+
+auth:
+  providers:
+    guest: {}
+
+catalog:
+  rules:
+    - allow:
+        - Component
+        - Resource
+        - API
+        - System
+        - Domain
+        - Location
+        - Template
+        - Group
+        - User
+
+boost:
+  security:
+    mode: 'development-only-no-auth'
+
+  model:
+    baseUrl: ${BOOST_MODEL_BASE_URL:-http://localhost:8080/v1}
+    name: ${BOOST_MODEL:-default}
+
+  providers:
+    llamastack:
+      baseUrl: ${LLAMASTACK_BASE_URL:-http://localhost:8321}
+
+    kagenti:
+      baseUrl: ${KAGENTI_BASE_URL:-http://localhost:8080}
+      namespaces:
+        - ${KAGENTI_NAMESPACE:-default}
+
+  kagenti:
+    auth:
+      tokenEndpoint: ${KAGENTI_TOKEN_ENDPOINT:-}
+      clientId: ${KAGENTI_CLIENT_ID:-}
+      clientSecret: ${KAGENTI_CLIENT_SECRET:-}
+
+  entityProviders:
+    kagenti:
+      baseUrl: ${KAGENTI_BASE_URL:-http://localhost:8080}
+      namespaces:
+        - ${KAGENTI_NAMESPACE:-default}
+      agentRefreshIntervalSeconds: 300
+      toolRefreshIntervalSeconds: 300

workspaces/boost/openspec/changes/platform-operations-deployment/specs/runtime-config/spec.md

@@ -103,13 +103,14 @@ The following new features MUST have runtime configuration fields as specified b
 
 #### Scenario: Token exchange configuration
 
-- **WHEN** the admin configures per-user Kagenti auth
+- **WHEN** the admin configures Kagenti service-account auth
 - **THEN** the following fields are available:
   | Field | Scope | Description |
   |---|---|---|
-  | `boost.kagenti.auth.tokenExchange.enabled` | yaml-only | Enable RFC 8693 token exchange |
-  | `boost.kagenti.auth.tokenExchange.audience` | yaml-only | Target audience for exchanged token |
-  | `boost.kagenti.auth.tokenExchange.userTokenHeader` | yaml-only | Header containing user OIDC token |
+  | `boost.kagenti.auth.tokenEndpoint` | yaml-only | Keycloak token endpoint URL |
+  | `boost.kagenti.auth.clientId` | yaml-only | OAuth2 client ID |
+  | `boost.kagenti.auth.clientSecret` | yaml-only | OAuth2 client secret (sensitive) |
+  | `boost.kagenti.auth.tokenExpiryBufferSeconds` | yaml-only | Seconds before expiry to refresh (default: 60) |
 
 #### Scenario: Credential encryption
 

workspaces/boost/openspec/changes/platform-operations-deployment/tasks.md

@@ -19,7 +19,7 @@
 - [ ] 2.2 Generate `config.d.ts` types from Zod schemas
 - [ ] 2.3 Validate all config writes (YAML and DB) via Zod `.parse()` — no hand-written validators
 - [ ] 2.4 Annotate each field with `configScope`: `yaml-only`, `db-overridable`, or `db-only`
-- [ ] 2.5 Add Zod schemas for new config fields: agentApproval, skillsMarketplace, tokenExchange, DevSpaces credentials
+- [ ] 2.5 Add Zod schemas for new config fields: agentApproval, skillsMarketplace, Keycloak service-account auth, DevSpaces credentials
 - [ ] 2.6 Admin UI shows only DB-overridable and DB-only fields
 - [ ] 2.7 Implement credential encryption for sensitive DB-stored values (DevSpaces tokens)
 - [ ] 2.8 Implement schema version tracking: store schema version alongside DB values, re-validate on startup

workspaces/boost/openspec/changes/pluggable-ai-platform-architecture/design.md

@@ -18,7 +18,7 @@ Boost implements the provider abstraction as modular RHDH dynamic plugins from t
 - Modifying chat interaction behavior or message rendering
 - Rewriting the ADK orchestration library
 - Creating catalog entities for models/agents (covered in agent-creation-discovery change)
-- Per-user token exchange (covered in security-safety-governance change)
+- Kagenti service-account auth (covered in security-safety-governance change — OAuth2 Client Credentials Grant adopted)
 
 ## Decisions
 

workspaces/boost/openspec/changes/pluggable-ai-platform-architecture/specs/provider-packaging/spec.md

@@ -61,7 +61,7 @@ The Kagenti provider MUST be packaged as an independent Backstage backend module
 
 #### Scenario: Keycloak token cache uses Backstage cacheService
 
-- **WHEN** the Kagenti module caches Keycloak access tokens (currently `KeycloakTokenManager`)
+- **WHEN** the Kagenti module caches Keycloak access tokens (currently `KeycloakAuthClient`)
 - **THEN** it uses `coreServices.cache` with TTL derived from token expiry
 - **AND** the token cache is Redis-backed in production for multi-instance safety
 

workspaces/boost/openspec/changes/security-safety-governance/design.md

@@ -4,13 +4,13 @@
 
 Boost builds the security and governance layer with Backstage fine-grained permissions as the sole authorization mechanism from day one. The Augment reference prototype's governance system grew into a parallel authorization layer where authorization decisions bypassed `permissions.authorize()`. Boost avoids this entirely.
 
-Boost also implements RFC 8693 token exchange for per-user identity delegation to Kagenti from the start, enabling per-user audit trails.
+Boost also implements OAuth2 Client Credentials Grant via `KeycloakAuthClient` for service-account authentication to Kagenti, with user identity propagated via `X-Backstage-User` header for audit trails.
 
 ## Goals
 
 - Implement 16 fine-grained Backstage permissions as the sole authorization mechanism
 - Add conditional permission rules for ownership (IS_OWNER), separation of duties (IS_NOT_CREATOR), and lifecycle stage gating (HAS_LIFECYCLE_STAGE)
-- Implement RFC 8693 token exchange for per-user Kagenti identity
+- Implement OAuth2 Client Credentials Grant for Kagenti service-account auth
 - Use `development-only-no-auth` as the only dev security mode name (no legacy aliases)
 - Add CSRF protection and credential encryption
 - Export all permissions from `boost-common`
@@ -40,20 +40,20 @@ These rules are evaluated against loaded resources via `createConditionalDecisio
 
 The `IS_NOT_CREATOR` permission rule is the primary enforcement mechanism. A route-level guard remains as defense-in-depth (belt and suspenders). Both layers are active in `security.mode === 'full'`.
 
-### Decision 4: Per-user token exchange is backend-only with graceful fallback
+### Decision 4: Service-account auth with KeycloakAuthClient
 
-`TokenExchangeManager` reads the user's OIDC token from a configurable request header (injected by auth proxy), exchanges it via RFC 8693, caches per-user, and deduplicates concurrent exchanges. All failures fall back silently to the shared service-account token — no request is ever blocked by token exchange issues. This is deliberately conservative: token exchange enhances audit trails and per-user authorization but must never degrade availability.
+`KeycloakAuthClient` acquires tokens via OAuth2 Client Credentials Grant, caches them with a configurable expiry buffer (`tokenExpiryBufferSeconds`, default: 60), and automatically refreshes before expiry. On 401 responses, the token is refreshed and the request retried once (max-1-retry). For user-initiated requests via `KagentiApiClient`, user identity is propagated via `X-Backstage-User` header for audit trails; entity provider background polling omits this header (no user context). This is deliberately simple: service-account auth provides consistent authentication without per-user token management complexity.
 
 ### Decision 5: Separation of authorization concerns
 
 Three non-overlapping authorization layers:
 
 - **Backstage** governs: UI visibility, agent lifecycle governance, ownership, approval workflows, admin operations
-- **Kagenti** governs: agent specs, tools, runtime operations — via per-user exchanged token when enabled
+- **Kagenti** governs: agent specs, tools, runtime operations — via service-account token with user identity in `X-Backstage-User` header
 - **Kubernetes** governs: pod/deployment operations, namespace scoping, SPIRE mTLS
 
 ## Risks
 
 - **RBAC policy complexity:** 16 permissions with conditions is more complex than 2. Mitigated by sensible defaults — `boost.access` as top-level gate and `boost.admin` available for coarse control.
-- **Token exchange reliability:** Keycloak availability becomes a dependency. Mitigated by graceful fallback to service-account token on any failure.
+- **Keycloak availability:** Keycloak becoming unavailable blocks Kagenti API calls. Mitigated by token caching with configurable expiry buffer, reducing the number of token requests.
 - **SonataFlow trust boundary:** Callbacks bypass self-approval prevention via header. Callback identity verification should be implemented to close this gap.

workspaces/boost/openspec/changes/security-safety-governance/proposal.md

@@ -21,8 +21,8 @@ Enterprise AI platforms must treat security, safety, and governance as foundatio
 ### Identity & Authentication
 
 - RBAC via Keycloak OIDC + Backstage permissions (`boost.access`, `boost.admin` as top-level gates)
-- RFC 8693 token exchange for per-user Kagenti identity delegation
-- Graceful fallback to service-account token on any exchange failure
+- OAuth2 Client Credentials Grant for Kagenti service-account authentication via `KeycloakAuthClient`
+- Token caching with configurable expiry buffer and max-1-retry on 401
 - MCP 4-level auth chain
 - Kagenti SPIRE integration for infrastructure mTLS
 
@@ -45,4 +45,4 @@ Enterprise AI platforms must treat security, safety, and governance as foundatio
 - `plugins/boost-common/src/permissions.ts` — 16 permission definitions with resource types
 - `plugins/boost-backend/src/middleware/security.ts` — `authorizeLifecycleAction` middleware
 - `plugins/boost-frontend/src/components/SecurityGate.tsx` — granular permission checks
-- `plugins/boost-backend/src/services/TokenExchangeManager.ts` — RFC 8693 implementation
+- `plugins/kagenti-entity-provider/src/providers/kagentiAuth.ts` — `KeycloakAuthClient` (OAuth2 Client Credentials Grant)

workspaces/boost/openspec/changes/security-safety-governance/specs/access-control/spec.md

@@ -81,41 +81,63 @@ Inference responses are not stored on the server when ZDR is enabled.
 
 ## ADDED Requirements
 
-### Requirement: Per-User Kagenti Identity via Token Exchange
+### Requirement: Service-Account Keycloak Authentication for Kagenti
 
-User identity MUST be delegated to Kagenti via RFC 8693 OAuth2 Token Exchange so agent operations are authorized per-user.
+Kagenti API calls MUST be authenticated via OAuth2 Client Credentials Grant using `KeycloakAuthClient` for service-account authentication. For user-initiated requests (chat, agent operations via `KagentiApiClient`), user identity is propagated via the `X-Backstage-User` header for audit purposes. Entity provider background polling has no user context and omits this header.
 
-#### Scenario: Token exchange enabled
+#### Scenario: Token acquisition
 
-- **WHEN** `boost.kagenti.auth.tokenExchange.enabled` is `true`
-- **AND** a user's OIDC token is available via the configured header (default: `X-Forwarded-Access-Token`)
-- **THEN** `TokenExchangeManager` exchanges the user's token for a Kagenti-scoped token via RFC 8693
-- **AND** the exchanged token is cached per-user with TTL from token expiry
-- **AND** concurrent exchanges for the same user are deduplicated
-- **AND** the per-user token is used for all Kagenti API calls on behalf of that user
+- **WHEN** `boost.kagenti.auth.tokenEndpoint`, `clientId`, and `clientSecret` are all configured
+- **THEN** `KeycloakAuthClient` acquires a bearer token via OAuth2 Client Credentials Grant
+- **AND** the token is cached until `expires_in - tokenExpiryBufferSeconds` seconds
+- **AND** the bearer token is included in all Kagenti API requests as `Authorization: Bearer <token>`
 
-#### Scenario: Token exchange graceful fallback
+#### Scenario: Streaming lifecycle with token refresh
 
-- **WHEN** token exchange fails (missing header, Keycloak error, exchange failure, disabled config)
-- **THEN** the system silently falls back to the shared service-account token
-- **AND** no request is blocked due to token exchange failure
-- **AND** the fallback is logged for debugging
+- **WHEN** a cached token is about to expire (within `tokenExpiryBufferSeconds`)
+- **THEN** a fresh token is acquired before the next API call
+- **AND** in-flight requests continue with the previously cached token
 
-#### Scenario: Token exchange configuration
+#### Scenario: 401 retry with max-1-retry constraint
 
-- **WHEN** token exchange is configured
-- **THEN** the following config is used:
+- **WHEN** a Kagenti API call returns HTTP 401
+- **THEN** the cached token is invalidated and a fresh token is acquired
+- **AND** the request is retried with the new token
+- **AND** if the retried request also returns 401, the error is propagated to the caller
+
+#### Scenario: User identity propagation (KagentiApiClient only)
+
+- **WHEN** a user-initiated Kagenti API call is made via `KagentiApiClient` (chat, agent operations)
+- **THEN** the `X-Backstage-User` header is set to the user's Backstage identity
+- **AND** the service-account bearer token is used for authentication (not the user's token)
+- **AND** entity provider background polling omits this header (no user context available)
+
+#### Scenario: Service-account auth configuration
+
+- **WHEN** Kagenti auth is configured
+- **THEN** the following config keys are used:
   | Key | Default | Description |
   |---|---|---|
-  | `boost.kagenti.auth.tokenExchange.enabled` | `false` | Enable per-user token exchange |
-  | `boost.kagenti.auth.tokenExchange.audience` | — | Target audience for exchanged token |
-  | `boost.kagenti.auth.tokenExchange.userTokenHeader` | `X-Forwarded-Access-Token` | Header containing user's OIDC token |
+  | `boost.kagenti.auth.tokenEndpoint` | — | Keycloak token endpoint URL |
+  | `boost.kagenti.auth.clientId` | — | OAuth2 client ID |
+  | `boost.kagenti.auth.clientSecret` | — | OAuth2 client secret (visibility: secret) |
+  | `boost.kagenti.auth.tokenExpiryBufferSeconds` | `60` | Seconds before expiry to refresh |
+
+#### Scenario: Kagenti REST API endpoints
+
+- **WHEN** fetching agents from Kagenti
+- **THEN** the URL pattern is `GET /api/v1/agents?namespace={ns}` (not `/a2a/`)
+- **AND** the response is unwrapped via `unwrapItems` to handle both `{ items: T[] }` and `T[]` shapes
+
+- **WHEN** fetching tools from Kagenti
+- **THEN** the URL pattern is `GET /api/v1/tools?namespace={ns}` (not `/a2a/`)
+- **AND** the response is unwrapped via `unwrapItems` to handle both response shapes
 
 #### Scenario: LlamaStack provider unaffected
 
-- **WHEN** token exchange is configured
+- **WHEN** Kagenti auth is configured
 - **THEN** `ResponsesApiProvider` is not modified — `setUserContext` is optional and not implemented
-- **AND** token exchange is Kagenti-specific
+- **AND** Keycloak auth is Kagenti-specific
 
 ### Requirement: CSRF Protection
 

workspaces/boost/openspec/changes/security-safety-governance/tasks.md

@@ -43,15 +43,15 @@
 - [ ] 6.4 Gate MCP panel with `boost.mcp.manage`
 - [ ] 6.5 Gate config panel with `boost.config.manage`
 
-## 7. Token Exchange (P1)
+## 7. Keycloak Service-Account Auth (P1)
 
-- [ ] 7.1 Create `TokenExchangeManager` implementing RFC 8693 exchange
-- [ ] 7.2 Add per-user token caching with TTL from token expiry
-- [ ] 7.3 Add concurrent exchange deduplication
-- [ ] 7.4 Add graceful fallback to service-account token on all failures
-- [ ] 7.5 Add config schema: `boost.kagenti.auth.tokenExchange.{enabled, audience, userTokenHeader}`
-- [ ] 7.6 Integrate into `KagentiApiClient.requestCore()` — inject per-user token when available
-- [ ] 7.7 Extract user OIDC token from configurable request header in route handlers
+- [x] 7.1 Create `KeycloakAuthClient` implementing OAuth2 Client Credentials Grant
+- [x] 7.2 Add token caching with configurable expiry buffer (`tokenExpiryBufferSeconds`, default: 60)
+- [ ] 7.3 Add max-1-retry on 401 (refresh token and retry once)
+- [x] 7.4 Add config schema: `boost.kagenti.auth.{tokenEndpoint, clientId, clientSecret, tokenExpiryBufferSeconds}`
+- [x] 7.5a Integrate into entity providers — inject bearer token (PR #3648)
+- [ ] 7.5b Integrate into `KagentiApiClient` — inject bearer token
+- [ ] 7.6 Propagate user identity via `X-Backstage-User` header for audit
 
 ## 8. CSRF and Credential Security (P2)
 
@@ -70,7 +70,7 @@
 - [ ] 10.2 Verify IS_OWNER blocks non-owner promote/delete/withdraw
 - [ ] 10.3 Verify IS_NOT_CREATOR blocks self-approval
 - [ ] 10.4 Verify `boost.admin` works as coarse-grained alternative to fine-grained permissions
-- [ ] 10.5 Verify token exchange fallback: disabled config → service-account token
-- [ ] 10.6 Verify token exchange fallback: Keycloak error → service-account token
-- [ ] 10.7 Verify token exchange fallback: missing header → service-account token
+- [ ] 10.5 Verify KeycloakAuthClient: token acquisition and caching
+- [ ] 10.6 Verify KeycloakAuthClient: max-1-retry on 401
+- [ ] 10.7 Verify entity providers include bearer token when auth is configured
 - [ ] 10.8 Verify `none` is rejected with a clear error pointing to `development-only-no-auth`

workspaces/boost/packages/backend/.eslintrc.js

@@ -0,0 +1 @@
+module.exports = require('@backstage/cli/config/eslint.backend');