fix(oauth): auto-learn IdP audience and persist as resource for token validation

[madhav165]

🐛 Bug-fix PR

📌 Summary

OAuth token audience validation fails for IdPs (ServiceNow, Authentik, etc.) that do not honor RFC 8707 and set the aud claim to an abstract identifier (e.g. client_id) rather than the resource URL sent in the authorization request. Users see "Token audience mismatch" errors after a successful OAuth flow, with no way to fix it from the UI.

RFC 8707 Section 2 explicitly allows this behavior:

The authorization server may use the exact resource value as the audience or it may map from that value to a more general URI or abstract identifier for the given resource.

🔁 Reproduction Steps

Closes #4384
Related: #4171

1. Register a gateway with OAuth (e.g. ServiceNow StreamableHTTP)
2. Complete OAuth authorization flow — succeeds
3. Click "Fetch Tools" — fails with: Token audience mismatch: token aud=[<client_id>], expected '<gateway_url>'

🐞 Root Cause

Two disconnected code paths:

1. /oauth/authorize and /oauth/callback (oauth_router.py) inject resource = _normalize_resource_url(gateway.url) into a transient copy of oauth_config and send it to the IdP. This copy is never persisted.

2. _validate_audience (token_validation_service.py:155) reads oauth_config.get("resource") from the persisted config (which has no resource), falls back to gateway_url, and fails because the IdP's aud doesn't match the gateway URL.

The system sends a resource to the IdP but never learns what the IdP actually mapped it to.

💡 Fix Description

Auto-learn the audience from the token at callback time and persist it as resource.

1. oauth_manager.py: Added _extract_token_audience() — decodes the access token (best-effort, no signature verification) and extracts the aud claim. Returns only the claim value, never the raw token. complete_authorization_code_flow now returns token_aud in its result dict.

2. oauth_router.py: Added _persist_learned_audience() — after complete_authorization_code_flow succeeds in the callback, persists the token_aud as resource in gateway.oauth_config. Skips gracefully for opaque tokens, missing aud, or when resource already matches.

3. oauth_router.py: Simplified resource injection in both /oauth/authorize and /oauth/callback — if resource is already set (learned from a previous token), use it as-is. If not set (first auth), use _normalize_resource_url(gateway.url).

4. token_validation_service.py: Simplified _validate_audience — expected = resource or gateway_url, normalize both sides to lists, simple membership check. Removed token audience and expected values from warning messages to avoid leaking sensitive identifiers (client IDs, etc.) into logs and error responses.

🧪 Verification

Check	Command	Status
Unit tests (6 affected files)	uv run pytest tests/unit/mcpgateway/routers/test_oauth_router.py tests/unit/mcpgateway/services/test_token_validation_service.py tests/unit/mcpgateway/services/test_gateway_service_oauth_comprehensive.py tests/unit/mcpgateway/test_oauth_manager.py tests/unit/mcpgateway/services/test_oauth_manager_pkce.py	✅ 385 passed
Manual regression (ServiceNow)	OAuth flow + Fetch Tools	✅ Fixed via DB update, code fix matches

📐 MCP Compliance (if relevant)

• Matches current MCP spec
• No breaking change to MCP clients

✅ Checklist

• Code formatted (make black isort pre-commit)
• No secrets/credentials committed
• Sensitive token values removed from log/error messages

[madhav165]

Self-Review Notes

Reviewed the following areas that are likely to come up in review — documenting findings here to avoid re-investigation:

1. Could _persist_learned_audience overwrite a user-configured resource?
No. There is no UI field or documented API parameter for setting resource in oauth_config. The admin UI (both create and edit forms) does not expose it, and the admin backend never assembles it. While oauth_config is a freeform Dict[str, Any] at the schema level, resource is exclusively auto-derived from gateway.url or auto-learned from the IdP's aud claim. Overwrite of user intent is not a realistic concern.

2. Should db.flush() in _persist_learned_audience be wrapped in try/except?
No. The codebase has 70+ db.flush() calls in production code, none of which are wrapped in try/except. The consistent pattern is to let flush failures propagate. Adding a guard here would be inconsistent with the rest of the codebase. If the DB is in a state where flush fails, that indicates a larger issue.

3. Is removal of pre-existing resource normalization a problem?
No. The fallback path (deriving resource from gateway.url) still goes through _normalize_resource_url(), which strips fragments and query per RFC 8707. Pre-existing/learned resource values are intentionally not normalized — they come from the IdP's aud claim and the client expects them verbatim. Re-normalizing an IdP-authoritative value would break the audience match this PR is fixing.

4. Are the redacted error messages in token_validation_service.py a debuggability concern?
The generic message ("token aud does not match expected resource or gateway URL") provides enough context to identify where to investigate without leaking client IDs or other sensitive identifiers into logs and error responses.

[MohanLaksh]

✅ Approve

Excellent fix for a critical OAuth bug. The solution is elegant and well-executed.

What's excellent:

• Smart approach: Auto-learning the IdP's actual aud claim at callback and persisting it solves the root disconnect cleanly
• RFC-compliant: Properly implements RFC 8707 Section 2's allowance for IdPs to map resource to different audiences
• Security-conscious: Removes sensitive token values from error messages (prevents client ID leaks)
• Code simplification: Replaced 20+ lines of complex resource normalization with simple learned-value logic
• Outstanding test coverage: 116 new test lines across 6 files, all 385 tests passing, full CI green (28/28 checks)
• Best-effort pattern: Gracefully handles opaque tokens and missing aud claims without breaking flows

Technical highlights:

• _extract_token_audience() does best-effort decode without signature verification (appropriate for learning vs. validation)
• _persist_learned_audience() is idempotent and respects transaction boundaries (flush not commit)
• Simplified _validate_audience() now handles both string and list forms cleanly

Verdict: Ready to merge.

This will unblock real-world OAuth deployments with ServiceNow, Authentik, and other IdPs that don't honor RFC 8707 strictly. Great work! 🎉

[brian-hussey]

Approved

[brian-hussey]

Approved, thank you for the submission.

[brian-hussey]

Failed tests are due to issue on main, I have validated all tests as part of this PR. Merging.