vMCP: add embedded authorization server

[tgrunnagle]

Overview

Add an optional embedded OAuth authorization server (AS) to vMCP. When enabled, the AS acts as the OIDC issuer for incoming clients: it drives users through one or more upstream IDPs, accumulates their tokens, and issues a ToolHive JWT. The OIDC middleware validates that TH-JWT via a self-referencing loopback and eagerly loads the upstream tokens into `identity.UpstreamTokens`. Existing behavior when no AS is configured (Mode A) is byte-for-byte unchanged.

Background

vMCP today requires clients to obtain upstream IDP tokens externally before connecting — there is no mechanism for vMCP to orchestrate the auth flow itself. Additionally, upstream tokens accumulated during login have no unified path into the outgoing auth layer; the `upstream_inject` strategy (which would inject an upstream provider token into backend requests) has no source to draw from.

This RFC adds Mode B, where vMCP hosts the auth server at the same origin, handles OAuth/OIDC discovery as a loopback, and makes upstream tokens available to outgoing auth strategies via `identity.UpstreamTokens`.

RFC: stacklok/toolhive-rfcs#53
Depends on: #3924 (RFC-0052 multi-upstream IDP support) for full end-to-end token flow

Task Breakdown

Task	GitHub Issue	Title	Depends On
Phase 1	#4140	Foundation — add AuthServerConfig model, CRD field, and structural validation	—
Phase 2	#4141	Server wiring — mount embedded auth server routes on vMCP mux	#4140
Phase 3	#4142	Startup validation (V-01..V-07) — upstream_inject types and validateAuthServerIntegration	#4140
Phase 4	#4143	Operator reconciler, HTTP handler unit tests, and E2E test coverage	#4141, #4142

#4140 (Phase 1: Foundation)
    |
    +---> #4141 (Phase 2: Server wiring)      \
    |                                          +---> #4143 (Phase 4: Operator + E2E)
    +---> #4142 (Phase 3: Startup validation) /

Phase 1 is the root; Phases 2 and 3 can be developed in parallel; Phase 4 is the integration phase.

Acceptance Criteria

• vMCP can be configured with an embedded AS (authServer.runConfig in YAML; spec.authServerConfigRef in K8s) and serves OAuth/OIDC endpoints at the same origin
• /.well-known/openid-configuration, /.well-known/oauth-authorization-server, /.well-known/jwks.json, and /oauth/ are mounted as unauthenticated routes in Mode B; Mode A is unaffected
• The /.well-known/ catch-all handler is replaced with an explicit /.well-known/oauth-protected-resource registration (non-breaking)
• Validation rules V-01..V-07 are enforced at startup (YAML path) and at operator reconciliation (K8s path), producing actionable error messages
• The operator reconciler resolves spec.authServerConfigRef, validates type and issuer consistency, and surfaces an AuthServerConfigValid status condition
• Mode A behavior is byte-for-byte identical to today — all new code paths are gated on cfg.AuthServer != nil
• All linked task issues (Phase 1: Foundation — add AuthServerConfig model, CRD field, and structural validation #4140, Phase 2: Server wiring — mount embedded auth server routes on vMCP mux #4141, Phase 3: Startup validation (V-01..V-07) — upstream_inject types and validateAuthServerIntegration #4142, Phase 4: Operator reconciler, HTTP handler unit tests, and E2E test coverage for embedded AS #4143) completed and merged
• Unit tests for validation rules (V-01..V-07), server route mounting (Mode A/B), and CRD structural validation
• E2E tests for Mode A/B routing, negative validation cases, and operator condition lifecycle

[tgrunnagle]

Planning Artifact: Intake

Intake: vMCP — Add Embedded Authorization Server (RFC-0053)

Original Requirements

Implement RFC-0053 (Embedded Auth Server in vMCP) as tracked by the placeholder epic issue
#4120. The RFC adds an optional embedded OAuth
authorization server to vMCP. When enabled, the AS acts as the OIDC issuer for incoming
clients: it drives users through one or more upstream IDPs, accumulates their tokens, and
issues a ToolHive JWT. Existing behavior for an external OIDC issuer is unchanged (byte-for-byte
backward compatibility).

Create exactly one GitHub issue per RFC implementation phase (4 phases total), linked as
sub-issues of #4120.

RFC document: /Users/trey/Documents/GitHub/toolhive/docs/proposals/THV-0053-vmcp-embedded-authserver.md

Existing Documentation

• RFC-0053: Embedded Auth Server in vMCP — /Users/trey/Documents/GitHub/toolhive/docs/proposals/THV-0053-vmcp-embedded-authserver.md — Full design: problem statement, high-level design (Mode A / Mode B), detailed config model changes, CRD changes, operator reconciler changes, HTTP route mounting, validation rules V-01..V-07, security considerations, testing strategy, and 4-phase implementation plan.
• RFC-0052 tracking issue — Auth Server: multi-upstream provider support #3924 — "Auth Server: multi-upstream provider support". Phases that depend on RFC-0052 (specifically identity.UpstreamTokens population by the OIDC auth middleware) should link to this issue.
• Epic placeholder issue — vMCP: add embedded authorization server #4120 — "vMCP: add embedded authorization server". All 4 phase issues are sub-issues of this epic.

Clarifying Questions & Answers

Issue Style

• Format: Task-based engineering tickets
• Rationale: Each phase maps 1:1 to a concrete set of file changes described in the RFC. Task-based framing (naming the files and actions) is more actionable than user-story framing for implementation work.

Repository & Project

• Source Code Repository: stacklok/toolhive (primary)
• Issue Repository: stacklok/toolhive (same)
• Existing Epic/Parent: vMCP: add embedded authorization server #4120 — vMCP: add embedded authorization server #4120
• Local Repository Clones:
  • stacklok/toolhive (primary — issues created here): /Users/trey/Documents/GitHub/toolhive — main service being changed; contains all affected packages (pkg/vmcp/, pkg/authserver/, cmd/vmcp/, cmd/thv-operator/)

Scope & Boundaries

• Expected Outcome: vMCP can optionally embed an OAuth authorization server that acts as the OIDC issuer for incoming clients, with full CRD support in the Kubernetes operator and startup validation for misconfigured auth wiring.
• In Scope:
  • Moving ExternalAuthConfigRef to its canonical location in mcpexternalauthconfig_types.go
  • Adding AuthServerConfigRef *ExternalAuthConfigRef to VirtualMCPServerSpec
  • Adding AuthServer *AuthServerConfig to pkg/vmcp/config/config.go
  • Regenerating zz_generated.deepcopy.go
  • Conditional AS creation and HTTP route mounting in cmd/vmcp/app/commands.go and pkg/vmcp/server/server.go
  • Replacing the /.well-known/ catch-all handler with explicit path registrations
  • Adding validateAuthServerIntegration with validation rules V-01 through V-07
  • Operator reconciler: resolving authServerConfigRef, cross-resource validation (type check, V-04, V-07), new AuthServerConfigValid status condition
  • CRD-to-config converter updates in cmd/thv-operator/pkg/vmcpconfig/converter.go
  • Unit tests (table-driven Ginkgo) for all validation rules
  • E2E tests for Mode A and Mode B behavior and negative validation cases
  • Documentation updates (docs/arch/09-operator-architecture.md, docs/arch/02-core-concepts.md, new vMCP auth server guide)
• Out of Scope:
  • upstream_inject outgoing auth strategy (deferred to a follow-up RFC)
  • Hot-reload of AS configuration
  • Multi-upstream IDP support (defined by RFC-0052, depended on but not re-specified here)
  • CLI-path upstream token swap (pkg/auth/upstreamswap middleware)
  • Step-up auth signaling
• Type: New feature (Mode B); non-breaking enhancement (Mode A preserved)

Technical Context

• Affected Components:
  • pkg/vmcp/config/config.go — new AuthServer *AuthServerConfig field and AuthServerConfig struct
  • pkg/vmcp/config/validator.go — new validateAuthServerIntegration method and helper functions
  • pkg/vmcp/config/zz_generated.deepcopy.go — regenerated
  • pkg/authserver/runner/embeddedauthserver.go — new RegisterHandlers(mux *http.ServeMux) method
  • cmd/vmcp/app/commands.go — conditional AS creation; pass AuthServer to server config
  • pkg/vmcp/server/server.go — add AuthServer to server Config; mount AS routes; replace /.well-known/ catch-all
  • cmd/thv-operator/api/v1alpha1/mcpserver_types.go — remove ExternalAuthConfigRef struct definition (moved)
  • cmd/thv-operator/api/v1alpha1/mcpexternalauthconfig_types.go — add ExternalAuthConfigRef struct definition (moved from mcpserver_types.go)
  • cmd/thv-operator/api/v1alpha1/virtualmcpserver_types.go — add AuthServerConfigRef field; add validateAuthServerConfig method
  • cmd/thv-operator/pkg/vmcpconfig/converter.go — resolve authServerConfigRef to AuthServerConfig.RunConfig
  • cmd/thv-operator/controllers/virtualmcpserver_controller.go — cross-resource validation, new status condition
• Known Constraints:
  • pkg/authserver/runner.EmbeddedAuthServer is already implemented; no new AS implementation work needed in this RFC
  • The /.well-known/ catch-all change is behavioral but non-breaking: the catch-all currently only serves oauth-protected-resource; all other paths return 404 in both old and new code
  • allowedAudiences is derived by the operator converter from the referencing VirtualMCPServer's incoming auth audience — it is NOT an explicit field on MCPExternalAuthConfig in the CRD
  • V-07 is an active startup check only on the YAML path; on the Kubernetes path it is satisfied by construction (converter derives the value)
  • Mode A (no AS) behavior must remain byte-for-byte identical; all new code paths are gated on cfg.AuthServer != nil
• Target Milestone: None specified

Dependencies & Integration

• Depends On:
  • RFC-0052 (multi-upstream IDP support) — Auth Server: multi-upstream provider support #3924 — Required for identity.UpstreamTokens population by the OIDC auth middleware. Phases 1–2 can be developed in parallel; full integration testing of the end-to-end flow requires RFC-0052 to be merged.
• Downstream Consumers:
  • upstream_inject outgoing auth strategy (deferred to a follow-up RFC) — will consume identity.UpstreamTokens populated by this RFC's auth middleware wiring
  • Cedar authorization policies — identity sub and claims will be derived from the first configured upstream IDP when Mode B is active
• External Integrations:
  • Upstream OIDC/OAuth IDPs configured by the operator (e.g., corporate SSO, GitHub)
  • Redis (optional AS session storage backend for production deployments)

Acceptance & Verification

• Success Criteria:
  • Mode A (no AS config): zero behavior change; all existing tests pass
  • Mode B (AS config present): /.well-known/openid-configuration returns a valid OIDC discovery document with the correct issuer and non-empty jwks_uri
  • Mode B: unauthenticated requests to the MCP endpoint return 401 (auth middleware remains active)
  • Validation rules V-01..V-07 fail fast at startup (YAML path) or at operator reconciliation (Kubernetes path) with descriptive error messages
  • Operator reconciler surfaces AuthServerConfigValid status condition on VirtualMCPServer
  • Invalid configurations produce a Failed phase with a descriptive condition message, preventing deployment until corrected
• Test Scenarios:
  • Unit (Ginkgo DescribeTable): one Entry per validation rule V-01..V-07, plus nil AuthServer (Mode A passes all checks) and a valid Mode B config
  • Unit: CRD validateAuthServerConfig — empty name rejection and nil-pointer safety
  • Unit: HTTP handler tests (Mode A: AS routes return 404; Mode B: AS routes served; oauth-protected-resource served in both modes)
  • E2E (test/e2e/vmcp_authserver_test.go): Mode B startup, OIDC discovery response, 401 on unauthenticated MCP request, Mode A 404 on discovery
  • E2E negative: issuer mismatch exits non-zero with V-04 error; upstream_inject with no AS exits with V-01 error; unknown provider exits with V-02 error
  • Operator E2E (Ginkgo/Gomega + Eventually): create MCPExternalAuthConfig + VirtualMCPServer, verify deployment and AuthServerConfigValid condition
• Stakeholders: tgrunnagle (RFC author, epic assignee)

Dry Run Mode

• Enabled: false — create issues directly in GitHub

Issue Labels

All phase issues inherit the epic's labels: authentication, authorization, enhancement, go, vmcp

Issue Linking

Each phase issue is linked to epic #4120 as a GitHub sub-issue using the gh sub-issue extension.

Assignees

Leave all phase issues unassigned.

Phase Breakdown

The RFC defines exactly 4 implementation phases. Each becomes one GitHub issue.

Phase 1: Foundation (no runtime behavior change)

Summary: Structural changes only — no runtime behavior changes. Move ExternalAuthConfigRef,
add new CRD and config fields, add structural validation, regenerate deepcopy.

File changes:

• cmd/thv-operator/api/v1alpha1/mcpserver_types.go — remove ExternalAuthConfigRef struct (moved)
• cmd/thv-operator/api/v1alpha1/mcpexternalauthconfig_types.go — add ExternalAuthConfigRef struct (moved from mcpserver_types.go); all existing callers are unaffected (same package)
• cmd/thv-operator/api/v1alpha1/virtualmcpserver_types.go — add AuthServerConfigRef *ExternalAuthConfigRef to VirtualMCPServerSpec; add validateAuthServerConfig() structural validation (name non-empty check)
• pkg/vmcp/config/config.go — add AuthServer *AuthServerConfig to Config; add AuthServerConfig struct wrapping authserver.RunConfig
• pkg/vmcp/config/zz_generated.deepcopy.go — regenerate

Acceptance: All existing tests pass. The new fields are optional and unused; no server behavior changes.

Dependencies: None (can start immediately)

---

Phase 2: Server wiring

Summary: Conditional AS creation at startup and HTTP route mounting. This is the first phase
that changes runtime behavior (when authServer is configured in the YAML).

File changes:

• pkg/authserver/runner/embeddedauthserver.go — add RegisterHandlers(mux *http.ServeMux) method mounting /oauth/, /.well-known/openid-configuration, /.well-known/oauth-authorization-server, /.well-known/jwks.json
• cmd/vmcp/app/commands.go — conditional AS creation (if cfg.AuthServer != nil); pass AuthServer to server config; hard failure on AS creation error (no silent fallback to Mode A)
• pkg/vmcp/server/server.go — add AuthServer *runner.EmbeddedAuthServer to server Config; call RegisterHandlers conditionally; replace mux.Handle("/.well-known/", ...) catch-all with explicit /.well-known/oauth-protected-resource registration

Smoke test: Start vMCP with a Mode B config; confirm /.well-known/openid-configuration returns AS metadata. Confirm Mode A behavior is unchanged.

Dependencies: Phase 1 must be merged. Can be developed in parallel with RFC-0052 (#3924), but full end-to-end auth flow requires RFC-0052.

---

Phase 3: Startup validation

Summary: Add validateAuthServerIntegration to the config validator, covering all V-01..V-07
rules, plus unit tests.

File changes:

• pkg/vmcp/config/validator.go — add validateAuthServerIntegration(cfg *Config) error called from Validate(); add helpers collectAllBackendStrategies, hasUpstreamProvider, containsString; extend validateBackendAuthStrategy for upstream_inject V-01/V-06

Tests:

• pkg/vmcp/config/validator_test.go — table-driven Ginkgo DescribeTable with one Entry per rule: V-01 (upstream_inject, no AS), V-02 (unknown provider), V-03 (token_exchange warning), V-04 (issuer mismatch), V-05 (AS internal validation failure), V-06 (empty provider name), V-07 (audience not in allowedAudiences); plus Mode A pass-through and valid Mode B config
• cmd/thv-operator/api/v1alpha1/virtualmcpserver_types_test.go — validateAuthServerConfig covering empty name rejection and nil-pointer safety

Dependencies: Phase 1 must be merged. Independent of Phase 2 (can be developed in parallel).

---

Phase 4: Operator reconciler

Summary: Kubernetes operator support — resolve authServerConfigRef, cross-resource
validation, new status condition, CRD converter updates, and E2E tests.

File changes:

• cmd/thv-operator/controllers/virtualmcpserver_controller.go — resolve spec.authServerConfigRef ref; verify type is embeddedAuthServer; verify issuer consistency (V-04); verify audience non-empty (defense-in-depth); surface AuthServerConfigValid status condition; invalid config produces Failed phase with descriptive message
• cmd/thv-operator/pkg/vmcpconfig/converter.go — resolve spec.authServerConfigRef → AuthServerConfig.RunConfig; derive allowedAudiences from spec.incomingAuth.oidcConfig.inline.audience

Tests:

• pkg/vmcp/server/server.go HTTP handler unit tests — Mode A: AS routes return 404, oauth-protected-resource returns RFC 9728 response; Mode B: AS routes served, oauth-protected-resource served, unauthenticated MCP catch-all returns 401
• test/e2e/vmcp_authserver_test.go — positive: Mode B OIDC discovery returns valid document; Mode B unauthenticated MCP returns 401; Mode A discovery returns 404
• test/e2e/vmcp_authserver_test.go — negative: issuer mismatch → non-zero exit + V-04 stderr; upstream_inject no AS → V-01 stderr; unknown provider → V-02 stderr
• Operator E2E (Ginkgo/Gomega, test/e2e/chainsaw/operator/ or test/e2e/): create MCPExternalAuthConfig + VirtualMCPServer; verify deployment; verify AuthServerConfigValid condition via Eventually

Documentation updates:

• docs/arch/09-operator-architecture.md — new VirtualMCPServerSpec.AuthServerConfigRef field and reconciliation steps
• docs/arch/02-core-concepts.md — Mode A / Mode B distinction for vMCP auth
• docs/ — new vMCP auth server guide (config walkthrough, example YAML, troubleshooting self-referencing OIDC discovery)
• Run task docs to regenerate CRD documentation

Dependencies: Phases 1, 2, and 3 must be merged. RFC-0052 (#3924) is required for full integration test of the end-to-end auth flow (upstream token population).

Additional Notes

• The EmbeddedAuthServer implementation in pkg/authserver/runner/ already exists and is used by the proxy runner. No new AS implementation is needed for this RFC — only the RegisterHandlers method is new.
• The self-referencing OIDC discovery loopback works because TokenValidator uses lazy OIDC discovery (deferred to first token validation) with exponential backoff retry. jwksAllowPrivateIP: true must be set in the incoming auth config for in-cluster deployments.
• V-07 (audience consistency) is checked only on the YAML path. On the Kubernetes path it is satisfied by construction: the converter derives allowedAudiences from the referencing VirtualMCPServer's incoming auth audience, so they are always consistent. The reconciler still checks post-conversion for defense-in-depth (empty audience guard).
• The ExternalAuthConfigRef move (Phase 1) is a struct definition move within the same Go package (v1alpha1). All existing callers using v1alpha1.ExternalAuthConfigRef are unaffected — no API surface changes.