Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions deployment/base/observability/gateway-telemetry-policy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ spec:
user: auth.identity.userid
# Subscription metadata for usage attribution and billing
subscription: auth.identity.selected_subscription
organization_id: auth.identity.organizationId
cost_center: auth.identity.costCenter
organization_id: auth.identity.subscription_info.organizationId
cost_center: auth.identity.subscription_info.costCenter
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
Expand Down
2 changes: 1 addition & 1 deletion docs/content/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ The MaaSAuthPolicy delegates to the MaaS API for key validation and subscription
2. MaaS API validates the key (format, not revoked, not expired) and returns username, groups, and subscription.
3. Authorino calls MaaS API to check subscription (groups, username, requested subscription from the key).
4. If the user lacks access to the requested subscription → error (403).
5. On success, returns selected subscription; Authorino caches the result (e.g., 60s TTL). AuthPolicy may inject `X-MaaS-Subscription` **server-side** for downstream rate limiting and metrics. Clients do not send this header on inference; subscription comes from the API key record created at mint time.
5. On success, returns selected subscription; Authorino caches the result (e.g., 60s TTL). Identity information (username, groups, subscription, key ID) is made available to TokenRateLimitPolicy and observability through AuthPolicy's `filters.identity` mechanism, but is **not forwarded** as HTTP headers to upstream model workloads (defense-in-depth security). Clients do not send subscription headers on inference; subscription comes from the API key record created at mint time.

```mermaid
graph TB
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,34 @@ The Kuadrant AuthPolicy validates API keys via the MaaS API and validates user t

---

## 9.1. Identity Headers and Defense-in-Depth

**For model inference routes** (HTTPRoutes targeting model workloads):

The controller-generated AuthPolicies do **not** inject most identity-related HTTP headers (`X-MaaS-Username`, `X-MaaS-Group`, `X-MaaS-Key-Id`) into requests forwarded to upstream model pods. This is a defense-in-depth security measure to prevent accidental disclosure of user identity, group membership, and key identifiers in:

- Model runtime logs
- Upstream debug dumps
- Misconfigured proxies or sidecars

**Exception:** `X-MaaS-Subscription` **is** injected for Istio Telemetry to enable per-subscription latency tracking. Istio runs in the Envoy gateway and cannot access Authorino's `auth.identity` context—it can only read request headers. The injected subscription value is server-controlled (resolved by Authorino from validated subscriptions), not client-provided.

All identity information remains available to **gateway-level features** through Authorino's `auth.identity` and `auth.metadata` contexts, which are consumed by:

- **TokenRateLimitPolicy (TRLP)**: Uses `selected_subscription_key`, `userid`, `groups`, and `subscription_info` from `filters.identity` (access `subscription_info.labels` for tier-based rate limiting)
- **Gateway telemetry/metrics**: Accesses identity fields with `metrics: true` enabled on `filters.identity`
- **Authorization policies**: OPA/Rego rules evaluate `auth.identity` and `auth.metadata` directly

**For maas-api routes**:

The static AuthPolicy for maas-api (`deployment/base/maas-api/policies/auth-policy.yaml`) still injects `X-MaaS-Username` and `X-MaaS-Group` headers, as maas-api's `ExtractUserInfo` middleware requires them. This is separate from model inference routes and follows a different security model (maas-api is a trusted internal service).

**Security motivation:**

Model workloads (vLLM, Llama.cpp, etc.) do not require strong identity claims in cleartext headers. By keeping identity at the gateway layer, we reduce the attack surface and limit the blast radius of potential log leaks or upstream vulnerabilities.

---

## 10. Summary

| Topic | Summary |
Expand Down
44 changes: 10 additions & 34 deletions maas-controller/pkg/controller/maas/maasauthpolicy_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -448,7 +448,11 @@ allow {
// match against subscription groups (which may differ from auth policy groups).
// Also inject subscription metadata from subscription-info for Limitador metrics.
// For API keys: username/groups come from apiKeyValidation metadata
// For K8s tokens: username/groups come from auth.identity
// Identity headers intentionally removed for defense-in-depth:
// User identity, groups, and key IDs are not forwarded to upstream model workloads
// to prevent accidental disclosure in logs or dumps. All identity information remains
// available to TRLP and telemetry via auth.identity and filters.identity below.
// Exception: X-MaaS-Subscription is injected for Istio Telemetry (per-subscription latency tracking).
rule["response"] = map[string]interface{}{
"success": map[string]interface{}{
"headers": map[string]interface{}{
Expand All @@ -463,31 +467,6 @@ allow {
"metrics": false,
"priority": int64(0),
},
// Username from API key validation or K8s token identity
"X-MaaS-Username": map[string]interface{}{
"plain": map[string]interface{}{
"expression": `(has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.username : auth.identity.user.username`,
},
"metrics": false,
"priority": int64(0),
},
// Groups - serialize to JSON array string from API key validation or K8s identity
// Using string() conversion of JSON-serialized groups for proper escaping
"X-MaaS-Group": map[string]interface{}{
"plain": map[string]interface{}{
"expression": `string(((has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.groups : auth.identity.user.groups))`,
},
"metrics": false,
"priority": int64(0),
},
// Key ID for tracking (only for API keys)
"X-MaaS-Key-Id": map[string]interface{}{
"plain": map[string]interface{}{
"expression": `(has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.keyId : ""`,
},
"metrics": false,
"priority": int64(0),
},
// Subscription bound to API key (only for API keys)
// For K8s tokens, this header is not injected (empty string)
"X-MaaS-Subscription": map[string]interface{}{
Expand Down Expand Up @@ -521,14 +500,11 @@ allow {
ref.Namespace, ref.Name,
),
},
"organizationId": map[string]interface{}{
"expression": `has(auth.metadata["subscription-info"].organizationId) ? auth.metadata["subscription-info"].organizationId : ""`,
},
"costCenter": map[string]interface{}{
"expression": `has(auth.metadata["subscription-info"].costCenter) ? auth.metadata["subscription-info"].costCenter : ""`,
},
"subscription_labels": map[string]interface{}{
"expression": `has(auth.metadata["subscription-info"].labels) ? auth.metadata["subscription-info"].labels : {}`,
// Full subscription-info object from subscription-select endpoint
// Contains: name, namespace, labels, organizationId, costCenter, error, message
// Consumers should access nested fields (e.g., subscription_info.organizationId)
"subscription_info": map[string]interface{}{
"expression": `has(auth.metadata["subscription-info"].name) ? auth.metadata["subscription-info"] : {}`,
},
// Error information (for debugging - only populated when selection fails)
"subscription_error": map[string]interface{}{
Expand Down
157 changes: 157 additions & 0 deletions maas-controller/pkg/controller/maas/maasauthpolicy_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1062,6 +1062,163 @@ func TestMaaSAuthPolicyReconciler_CacheKeyModelIsolation(t *testing.T) {
}
}

// TestMaaSAuthPolicyReconciler_NoIdentityHeadersUpstream verifies that identity headers
// (X-MaaS-Username, X-MaaS-Group, X-MaaS-Key-Id) are NOT injected into requests forwarded
// to upstream model workloads (defense-in-depth). Exception: X-MaaS-Subscription IS injected
// for Istio Telemetry (per-subscription latency tracking).
// All identity information remains available to TRLP and telemetry via filters.identity.
func TestMaaSAuthPolicyReconciler_NoIdentityHeadersUpstream(t *testing.T) {
const (
modelName = "llm"
namespace = "default"
httpRouteName = "maas-model-" + modelName
authPolicyName = "maas-auth-" + modelName
maasPolicyName = "policy-a"
)

model := newMaaSModelRef(modelName, namespace, "ExternalModel", modelName)
route := newHTTPRoute(httpRouteName, namespace)
maasPolicy := newMaaSAuthPolicy(maasPolicyName, namespace, "team-a", maasv1alpha1.ModelRef{Name: modelName, Namespace: namespace})

c := fake.NewClientBuilder().
WithScheme(scheme).
WithRESTMapper(testRESTMapper()).
WithObjects(model, route, maasPolicy).
WithStatusSubresource(&maasv1alpha1.MaaSAuthPolicy{}).
Build()

r := &MaaSAuthPolicyReconciler{
Client: c,
Scheme: scheme,
MaaSAPINamespace: "maas-system",
MetadataCacheTTL: 60,
AuthzCacheTTL: 60,
}

req := ctrl.Request{NamespacedName: types.NamespacedName{Name: maasPolicyName, Namespace: namespace}}
if _, err := r.Reconcile(context.Background(), req); err != nil {
t.Fatalf("Reconcile: unexpected error: %v", err)
}

got := &unstructured.Unstructured{}
got.SetGroupVersionKind(schema.GroupVersionKind{Group: "kuadrant.io", Version: "v1", Kind: "AuthPolicy"})
if err := c.Get(context.Background(), types.NamespacedName{Name: authPolicyName, Namespace: namespace}, got); err != nil {
t.Fatalf("Get AuthPolicy %q: %v", authPolicyName, err)
}

// Test 1: Verify identity headers (except X-MaaS-Subscription) are not forwarded upstream
t.Run("identity headers not forwarded to upstream", func(t *testing.T) {
headers, found, err := unstructured.NestedMap(got.Object, "spec", "rules", "response", "success", "headers")
if err != nil {
t.Fatalf("Error checking headers: %v", err)
}

if !found {
t.Fatalf("response.success.headers should exist (X-MaaS-Subscription for Istio)")
}

// Verify X-MaaS-Subscription IS present (required for Istio Telemetry)
if _, exists := headers["X-MaaS-Subscription"]; !exists {
t.Errorf("X-MaaS-Subscription header should be present for Istio Telemetry")
}

// Verify identity headers are NOT present (defense-in-depth)
forbiddenHeaders := []string{"X-MaaS-Username", "X-MaaS-Group", "X-MaaS-Key-Id"}
for _, header := range forbiddenHeaders {
if _, exists := headers[header]; exists {
t.Errorf("Identity header %q should NOT be forwarded to upstream workloads (defense-in-depth)", header)
}
}
})

// Test 2: Verify filters.identity exists and contains all necessary data for TRLP and telemetry
t.Run("identity data available for TRLP and telemetry", func(t *testing.T) {
identity, found, err := unstructured.NestedMap(got.Object, "spec", "rules", "response", "success", "filters", "identity", "json", "properties")
if err != nil || !found {
t.Fatalf("filters.identity.json.properties missing: found=%v err=%v", found, err)
}

// Verify essential fields for TRLP
requiredFields := []string{
"userid", // User identification
"groups", // User groups
"selected_subscription_key", // Model-scoped subscription key for TRLP
"selected_subscription", // Subscription name
"subscription_info", // Full subscription object (includes labels, organizationId, costCenter, etc.)
}

for _, field := range requiredFields {
if _, exists := identity[field]; !exists {
t.Errorf("filters.identity must include %q for TRLP/telemetry, but it's missing", field)
}
}

// Verify telemetry/observability fields
observabilityFields := []string{
"keyId", // API key tracking
}

for _, field := range observabilityFields {
if _, exists := identity[field]; !exists {
t.Errorf("filters.identity should include %q for observability, but it's missing", field)
}
}

// Verify subscription_info contains full object (not just individual fields)
// This object should contain: name, namespace, labels, organizationId, costCenter
// Consumers should access nested fields like:
// - subscription_info.organizationId (for billing/cost attribution)
// - subscription_info.costCenter (for chargeback)
// - subscription_info.labels (for tier-based rate limiting)
subscriptionInfoField, exists := identity["subscription_info"]
if !exists {
t.Error("filters.identity must include 'subscription_info' field (full object from subscription-select)")
} else {
// Verify it's an expression that returns the full object
subscriptionInfoMap, ok := subscriptionInfoField.(map[string]interface{})
if !ok {
t.Errorf("subscription_info should be a map, got %T", subscriptionInfoField)
} else {
expr, hasExpr := subscriptionInfoMap["expression"]
if !hasExpr {
t.Error("subscription_info should have an 'expression' field")
} else {
exprStr, ok := expr.(string)
if !ok {
t.Errorf("subscription_info expression should be a string, got %T", expr)
} else {
// Verify it returns the full object, not just a sub-field
if !contains(exprStr, `auth.metadata["subscription-info"]`) {
t.Errorf("subscription_info expression should reference full subscription-info object, got: %s", exprStr)
}
// Verify it doesn't extract only labels (old behavior)
if contains(exprStr, ".labels") && !contains(exprStr, "?") {
t.Errorf("subscription_info expression should return full object, not just .labels, got: %s", exprStr)
}
// Note: We can't verify organizationId and costCenter exist at runtime here
// (they're optional fields that depend on MaaSSubscription configuration)
// but the full object is passed, so consumers can access them via:
// auth.identity.subscription_info.organizationId
// auth.identity.subscription_info.costCenter
}
}
}
}
})

// Test 3: Verify metrics are enabled on identity filter
t.Run("identity filter has metrics enabled", func(t *testing.T) {
metricsEnabled, found, err := unstructured.NestedBool(got.Object, "spec", "rules", "response", "success", "filters", "identity", "metrics")
if err != nil || !found {
t.Fatalf("filters.identity.metrics missing: found=%v err=%v", found, err)
}

if !metricsEnabled {
t.Error("filters.identity.metrics must be true for telemetry to access identity data")
}
})
}

// contains is a helper to check if a string contains a substring (case-sensitive).
func contains(s, substr string) bool {
return len(s) >= len(substr) && (s == substr || len(substr) == 0 ||
Expand Down
Loading