chore: upgrade/validate Kuadrant 1.4.2 (opendatahub-io#643)

# Upgrade to Kuadrant 1.4.2
https://redhat.atlassian.net/browse/RHOAIENG-55749
https://redhat.atlassian.net/browse/RHOAIENG-54157
## Summary
Upgrades deployment to use Kuadrant 1.4.2 (from 1.3.1) to enable
authorization header stripping for security and migrates AuthPolicy
configurations to the new schema format required
  by Authorino v0.23.1.                                     
## Changes
   
### 1. Kuadrant Upgrade
- Update `scripts/deploy.sh` to install Kuadrant v1.4.2 catalog (was
v1.3.1)
- Brings Authorino v0.23.1 (was v0.22.0)
- **Required for authorization header stripping capability**
### 2. Authorization Header Stripping for Internal Models
**Security enhancement to prevent credential exfiltration**
Added authorization header stripping in
`maas-controller/pkg/controller/maas/maasauthpolicy_controller.go` for
internal KServe models.
**Problem:** User credentials (OpenShift tokens or API keys) were being
forwarded to model backends, creating a credential exfiltration risk
where malicious or compromised models
  could capture and misuse tokens.                          
**Solution:** Authorino now strips the Authorization header before
forwarding requests to internal model backends:
   
```go
  rule["response"] = map[string]any{                        
"success": map[string]any{
"headers": map[string]any{
// Strip Authorization header to prevent token exfiltration
"Authorization": map[string]any{
"plain": map[string]any{
"value": "",
},
"key": "authorization",
"metrics": false,
"priority": int64(0),
},
// ... other headers (username, groups, subscription)
Impact:
- User tokens and API keys are validated by Authorino but NOT forwarded
to model services
- Applies to both inference requests and model discovery probing
- Only affects internal KServe models (ExternalModel resources with
credentialRef are unaffected)
Version requirement verified: Testing confirmed that Kuadrant 1.4.1 /
Authorino v0.24.0 does not properly strip headers (appends empty value
but leaves original token present).
Kuadrant 1.4.2 / Authorino v0.23.1 correctly replaces the authorization
header.
3. AuthPolicy Schema Migration
                                                            
Migrate deployment/base/maas-api/policies/auth-policy.yaml from CEL
predicate format to selector/operator format:
                                                            
Before (v1.3.1 format):
  when:                                                     
- predicate: request.headers.authorization.startsWith("Bearer sk-oai-")
After (v1.4.2 format):
when:
- selector: request.headers.authorization
operator: matches
value: "^Bearer sk-oai-.*"
4. Fallback Header Handling
- Removed negative lookahead regex from fallback headers
(X-MaaS-Username-OC, X-MaaS-Group-OC)
- Authorino v0.23.1's matches operator doesn't support ^Bearer
(?!sk-oai-).* pattern
- Now rely on priority system instead:
- API key headers: priority: 0 (higher priority, evaluated first)
- Fallback headers: priority: 1 (lower priority, only used when API key
when clause doesn't match)
5. Subscription Header Fix
Fixed handling of multiple X-Maas-Subscription header values in
maas-api/internal/handlers/models.go:
                                                            
Problem: When clients send x-maas-subscription header (even empty
string), Authorino appends its injected value, resulting in:
  X-Maas-Subscription: ["", "simulator-subscription"]       
Gin's GetHeader() returns only the first value (empty string), causing
403 errors.
Solution: Iterate header values in reverse order and take the last
non-empty value:
headerValues := c.Request.Header.Values("X-Maas-Subscription")
for i := len(headerValues) - 1; i >= 0; i-- {
trimmed := strings.TrimSpace(headerValues[i])
if trimmed != "" {
requestedSubscription = trimmed
break
}
}
This ensures we use Authorino's validated subscription when available,
while still supporting clients that don't send the header.
6. Documentation Updates
Updated version requirements in README.md and
docs/content/install/prerequisites.md:
  - MaaS v0.2.0+ requires Kuadrant 1.4.2+ (ODH) or RHCL 1.3+ (RHOAI)
- Added detailed explanation of authorization header stripping security
requirement
                                                            
Testing
                                                            
All E2E tests passing (74 passed):
                                                            
Fixed by subscription header change:
- ✅ test_empty_subscription_header_value - correctly auto-selects
subscription when empty header sent
- ✅ test_api_key_ignores_subscription_header - correctly uses API key's
bound subscription, ignoring client header
                                                            
Verified working:
  - ✅ API key authentication with subscription binding     
- ✅ OpenShift token authentication
  - ✅ Subscription-scoped model filtering                  
- ✅ Rate limiting with TokenRateLimitPolicy
- ✅ Namespace-scoped resource access
- ✅ Cross-namespace model references
Authorization header stripping verified:
- ✅ Kuadrant 1.4.2 / Authorino v0.23.1: Header correctly stripped
(backend receives empty string)
- ❌ Kuadrant 1.4.1 / Authorino v0.24.0: Header NOT properly stripped
(original token still present)
Compatibility
                                                            
The new AuthPolicy schema format is backward compatible with Kuadrant
1.3.1. This was verified by:
  1. Deploying main branch code (predicate format) with Kuadrant 1.4.2 ✅
2. Deploying new schema format with Kuadrant 1.3.1 ✅
                                                            
This allows safe deployment of auth-policy changes before operator
upgrade.
Migration Notes
For clusters upgrading from Kuadrant 1.3.x to 1.4.x:
   
1. Apply updated AuthPolicy first (backward compatible)
  2. Upgrade Kuadrant operator to 1.4.2                     
3. Monitor Authorino reconciliation - should be seamless
4. Validate auth flows - API key and OpenShift token authentication
5. Verify header stripping - User tokens should not reach model backends
No service disruption expected during upgrade.
Related Issues
                                                            
- [RHOAIENG-55749](https://redhat.atlassian.net/browse/RHOAIENG-55749):
Validate Kuadrant/RHCL Compatibility and Upgrade
- [RHOAIENG-54157](https://redhat.atlassian.net/browse/RHOAIENG-54157):
Strip Authorization Header to Prevent Credential Exfiltration
Merge criteria:
- The commits are squashed in a cohesive manner and have meaningful
messages.
- Testing instructions have been added in the PR body (for PRs involving
changes that are not immediately obvious).
- The developer has manually tested the changes and verified that the
changes work
Summary by CodeRabbit
- Security
- Added authorization header stripping for internal models to prevent
credential exfiltration to model backends
- Bug Fixes
- Improved API key validation logic for more robust authentication
handling
- Enhanced subscription header processing to correctly handle multiple
header instances
- Chores
- Upgraded Kuadrant policy-engine dependency from v1.3.1 to v1.4.2
- Documentation
- Updated version requirements to specify Kuadrant 1.4.2+ / RHCL 1.3+
for MaaS v0.2.0+
- Added security context explanation for authorization header stripping
requirement

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Strengthened API-key checks using regex-based matching and adjusted
header precedence so API-key-derived values override fallback headers.
* Fixed subscription header handling to pick the last non-empty value
when multiple headers are present and tightened non-empty checks.
* Ensure upstream Authorization is stripped from responses to prevent
credential forwarding.

* **Documentation**
* Updated prerequisites and migration notes; clarified minimum versions.

* **Chores**
  * Bumped Kuadrant policy-engine reference to v1.4.2.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

Author: jrhyness

README.md

@@ -16,15 +16,44 @@ Our goal is to create a comprehensive platform for **Models as a Service** with
 
 ## 📋 Prerequisites
 
-- **Openshift cluster** (4.19.9+) with kubectl/oc access
+- **OpenShift cluster** (4.19.9+) with kubectl/oc access
+- **Kuadrant v1.4.2+** (ODH) or **RHCL v1.3+** (RHOAI) - **Required for MaaS v0.2.0+**
 - **PostgreSQL database** (for production ODH/RHOAI deployments)
 
-!!! warning "Database Required for Production"
-    MaaS requires a PostgreSQL database for API key management. For production ODH/RHOAI deployments, you must create a Secret with the database connection URL **before** enabling modelsAsService.
+### ⚠️ Important Version Requirements
 
-    See [Database Prerequisites](docs/content/install/prerequisites.md#database-prerequisite) for details.
+#### Kuadrant 1.4.2+ Required (MaaS v0.2.0+)
 
-    Note: The `scripts/deploy.sh` script creates a development PostgreSQL instance automatically.
+**MaaS v0.2.0 and later requires Kuadrant 1.4.2+ (ODH) or RHCL 1.3+ (RHOAI).**
+
+**Why Kuadrant 1.4.2+ is required:**
+
+MaaS v0.2.0 requires the authorization header stripping capability added in Authorino v0.23.1 (shipped with Kuadrant 1.4.2) to protect user credentials from potential exfiltration to model backends.
+
+**Security Context:**
+
+When a user makes an inference request with their OpenShift token or API key, that credential must be validated by Authorino but should NOT be forwarded to model backends (whether internal KServe models or external providers). Kuadrant 1.4.2+ allows Authorino to:
+
+1. Validate the incoming user credential (OpenShift token or MaaS API key)
+2. Strip/replace the Authorization header before forwarding to model backends
+3. Optionally inject model-specific credentials from Kubernetes Secrets (credentialRef) for ExternalModel resources
+
+This prevents credential exfiltration where a malicious or compromised model service could capture and misuse user tokens.
+
+**Migration Notes:**
+
+- The deployment script (`scripts/deploy.sh`) automatically installs Kuadrant 1.4.2 for new deployments
+- For existing deployments, upgrade Kuadrant/RHCL before upgrading to MaaS v0.2.0+
+
+For detailed version compatibility, see [Version Compatibility](docs/content/install/prerequisites.md#version-compatibility).
+
+#### Database Required for Production
+
+MaaS requires a PostgreSQL database for API key management. For production ODH/RHOAI deployments, you must create a Secret with the database connection URL **before** enabling modelsAsService.
+
+See [Database Prerequisites](docs/content/install/prerequisites.md#database-prerequisite) for details.
+
+Note: The `scripts/deploy.sh` script creates a development PostgreSQL instance automatically.
 
 ## 🚀 Quick Start
 

deployment/base/maas-api/policies/auth-policy.yaml

@@ -12,7 +12,9 @@ spec:
       # API key authentication (for sk-oai-* tokens)
       api-keys:
         when:
-          - predicate: request.headers.authorization.startsWith("Bearer sk-oai-")
+          - selector: request.headers.authorization
+            operator: matches
+            value: "^Bearer sk-oai-.*"
         plain:
           selector: request.headers.authorization
         priority: 0
@@ -27,7 +29,9 @@ spec:
       # Validate API key via HTTP callback (only runs for API key auth)
       apiKeyValidation:
         when:
-          - predicate: request.headers.authorization.startsWith("Bearer sk-oai-")
+          - selector: request.headers.authorization
+            operator: matches
+            value: "^Bearer sk-oai-.*"
         http:
           # Placeholder URL - gets patched based on deployment mode:
           #   - Operator mode (ODH/RHOAI): ODH overlay replacement (app-namespace param)
@@ -42,7 +46,9 @@ spec:
       # Check API key is valid (only for API key auth)
       api-key-valid:
         when:
-          - predicate: request.headers.authorization.startsWith("Bearer sk-oai-")
+          - selector: request.headers.authorization
+            operator: matches
+            value: "^Bearer sk-oai-.*"
         patternMatching:
           patterns:
             - selector: auth.metadata.apiKeyValidation.valid
@@ -55,29 +61,31 @@ spec:
           # Username: from API key validation (when API key used)
           X-MaaS-Username:
             when:
-              - predicate: request.headers.authorization.startsWith("Bearer sk-oai-")
+              - selector: request.headers.authorization
+                operator: matches
+                value: "^Bearer sk-oai-.*"
             plain:
               selector: auth.metadata.apiKeyValidation.username
             priority: 0
-          # Username: from OpenShift identity (when OC token used)
+          # Username: from OpenShift identity (fallback for non-API-key tokens)
+          # No 'when' clause - always tries to inject, but priority 1 means API key wins
           X-MaaS-Username-OC:
-            when:
-              - predicate: '!request.headers.authorization.startsWith("Bearer sk-oai-")'
             plain:
               selector: auth.identity.user.username
             key: X-MaaS-Username
             priority: 1
           # Groups: from API key validation as JSON array (when API key used)
           X-MaaS-Group:
             when:
-              - predicate: request.headers.authorization.startsWith("Bearer sk-oai-")
+              - selector: request.headers.authorization
+                operator: matches
+                value: "^Bearer sk-oai-.*"
             plain:
               selector: auth.metadata.apiKeyValidation.groups.@tostr
             priority: 0
-          # Groups: from OpenShift identity as JSON array (when OC token used)
+          # Groups: from OpenShift identity as JSON array (fallback for non-API-key tokens)
+          # No 'when' clause - always tries to inject, but priority 1 means API key wins
           X-MaaS-Group-OC:
-            when:
-              - predicate: '!request.headers.authorization.startsWith("Bearer sk-oai-")'
             plain:
               selector: auth.identity.user.groups.@tostr
             key: X-MaaS-Group
@@ -86,8 +94,12 @@ spec:
           # This header is used by /v1/models to determine which subscription's models to return
           X-MaaS-Subscription:
             when:
-              - predicate: request.headers.authorization.startsWith("Bearer sk-oai-")
-              - predicate: auth.metadata.apiKeyValidation.subscription != ""
+              - selector: request.headers.authorization
+                operator: matches
+                value: "^Bearer sk-oai-.*"
+              - selector: auth.metadata.apiKeyValidation.subscription
+                operator: neq
+                value: ""
             plain:
               selector: auth.metadata.apiKeyValidation.subscription
             priority: 0

docs/content/install/prerequisites.md

@@ -12,6 +12,7 @@ Red Hat OpenShift AI (RHOAI). MaaS is installed by enabling it in the DataScienc
 |--------------|-----|-------------------------------|-------------|
 | v0.0.2       | 4.19.9+ | v1.3+ / v1.2+             | v1.2+       |
 | v0.1.0       | 4.19.9+ | v1.3+ / v1.2+             | v1.2+       |
+| v0.2.0+      | 4.19.9+ | v1.4.2+ / v1.3+           | v1.2+       |
 
 !!! note "Other Kubernetes flavors"
     Other Kubernetes flavors (e.g., upstream Kubernetes, other distributions) are currently being validated.
@@ -34,14 +35,12 @@ MaaS requires Open Data Hub version 3.0 or later, with the Model Serving compone
 enabled (KServe) and properly configured for deploying models with `LLMInferenceService`
 resources.
 
-A specific requirement for MaaS is to set up ODH's Model Serving with Kuadrant v1.3+, even
-though ODH can work with earlier Kuadrant versions.
+A specific requirement for MaaS v0.2.0+ is to set up ODH's Model Serving with Kuadrant v1.4.2 or later.
 
 ## Requirements for Red Hat OpenShift AI
 
 MaaS requires Red Hat OpenShift AI (RHOAI) version 3.0 or later, with the Model Serving
 component enabled (KServe) and properly configured for deploying models with
 `LLMInferenceService` resources.
 
-A specific requirement for MaaS is to set up RHOAI Model Serving with Red Hat Connectivity
-Link (RHCL) v1.2+, even though RHOAI can work with earlier RHCL versions.
+A specific requirement for MaaS v0.2.0+ is to set up RHOAI Model Serving with Red Hat Connectivity Link (RHCL) v1.3 or later.

maas-api/internal/handlers/models.go

@@ -187,7 +187,17 @@ func (h *ModelsHandler) ListLLMs(c *gin.Context) {
 	// Extract x-maas-subscription header.
 	// For API keys: Authorino injects this from auth.metadata.apiKeyValidation.subscription
 	// For user tokens: This header is not present (Authorino doesn't inject it)
-	requestedSubscription := strings.TrimSpace(c.GetHeader("x-maas-subscription"))
+	// Note: If client sends x-maas-subscription header, there may be multiple values.
+	// Authorino appends its value, so we take the last non-empty value.
+	requestedSubscription := ""
+	headerValues := c.Request.Header.Values("X-Maas-Subscription")
+	for i := len(headerValues) - 1; i >= 0; i-- {
+		trimmed := strings.TrimSpace(headerValues[i])
+		if trimmed != "" {
+			requestedSubscription = trimmed
+			break
+		}
+	}
 	isAPIKeyRequest := strings.HasPrefix(authHeader, "Bearer sk-oai-")
 
 	// Fail closed: API keys without a bound subscription must be rejected

maas-controller/pkg/controller/maas/maasauthpolicy_controller.go

@@ -452,6 +452,17 @@ allow {
 		rule["response"] = map[string]interface{}{
 			"success": map[string]interface{}{
 				"headers": map[string]interface{}{
+					// Strip Authorization header to prevent token exfiltration to model backends
+					// Both API keys and OpenShift tokens are validated by Authorino, but should
+					// not be forwarded to model services to prevent credential theft
+					"Authorization": map[string]interface{}{
+						"plain": map[string]interface{}{
+							"value": "",
+						},
+						"key":      "authorization",
+						"metrics":  false,
+						"priority": int64(0),
+					},
 					// Username from API key validation or K8s token identity
 					"X-MaaS-Username": map[string]interface{}{
 						"plain": map[string]interface{}{

scripts/deploy.sh

@@ -11,7 +11,7 @@
 # OPTIONS:
 #   --operator-type <odh|rhoai>   Operator to install (default: odh)
 #                                 Policy engine is auto-selected:
-#                                   odh → kuadrant (community v1.3.1)
+#                                   odh → kuadrant (community v1.4.2)
 #                                   rhoai → rhcl (Red Hat Connectivity Link)
 #   --enable-tls-backend          Enable TLS for Authorino/MaaS API (default: on)
 #   --enable-keycloak             Deploy Keycloak for external OIDC (optional)
@@ -114,7 +114,7 @@ OPTIONS:
       Which operator to install (default: odh)
       Policy engine is auto-selected based on operator type:
       - rhoai → rhcl (Red Hat Connectivity Link)
-      - odh → kuadrant (community v1.3.1 with AuthPolicy v1)
+      - odh → kuadrant (community v1.4.2 with AuthPolicy v1)
       Only applies when --deployment-mode=operator
 
   --enable-tls-backend
@@ -372,13 +372,13 @@ validate_configuration() {
   fi
 
   # Auto-determine policy engine based on operator type
-  # - ODH uses community Kuadrant (v1.3.1 from upstream catalog has AuthPolicy v1)
+  # - ODH uses community Kuadrant (v1.4.2 from upstream catalog has AuthPolicy v1)
   # - RHOAI uses RHCL (Red Hat Connectivity Link - downstream)
   if [[ "$DEPLOYMENT_MODE" == "operator" ]]; then
     case "$OPERATOR_TYPE" in
       odh)
         POLICY_ENGINE="kuadrant"
-        log_debug "Auto-selected policy engine for ODH: kuadrant (community v1.3.1)"
+        log_debug "Auto-selected policy engine for ODH: kuadrant (community v1.4.2)"
         ;;
       rhoai)
         POLICY_ENGINE="rhcl"
@@ -866,14 +866,14 @@ install_policy_engine() {
       ;;
 
     kuadrant)
-      log_info "Installing Kuadrant v1.3.1 (upstream community)"
+      log_info "Installing Kuadrant v1.4.2 (upstream community)"
 
-      # Create custom catalog for upstream Kuadrant v1.3.1
-      # This version provides AuthPolicy v1 API required by ODH
+      # Create custom catalog for upstream Kuadrant v1.4.2
+      # This version provides AuthPolicy v1 API and Authorino v0.23.1
       local kuadrant_catalog="kuadrant-operator-catalog"
       local kuadrant_ns="kuadrant-system"
 
-      log_info "Creating Kuadrant v1.3.1 catalog source..."
+      log_info "Creating Kuadrant v1.4.2 catalog source..."
       kubectl create namespace "$kuadrant_ns" 2>/dev/null || true
 
       cat <<EOF | kubectl apply -f -
@@ -884,7 +884,7 @@ metadata:
   namespace: $kuadrant_ns
 spec:
   sourceType: grpc
-  image: quay.io/kuadrant/kuadrant-operator-catalog:v1.3.1
+  image: quay.io/kuadrant/kuadrant-operator-catalog:v1.4.2
   displayName: Kuadrant Operator Catalog
   publisher: Kuadrant
   updateStrategy: